Methods and apparatus for employing a user&#39;s location in providing information to the user

ABSTRACT

Some embodiments relate to techniques for receiving a query from a device and location data indicating a location of the device, the location data having a level of specificity; in response to the query being received, identifying at least one first search engine to which to submit a representation of the query; determining whether the level of specificity of the received location data is sufficient for the at least one first search engine; if the level of specificity of the location data is sufficient, instructing the device to issue the representation of the query to the at least one first search engine; and if the level of specificity of the location data is not sufficient, instructing the device to send, to the at least one computer, location data at a greater level of specificity.

BACKGROUND

1. Field of Invention

The techniques described herein are directed generally to the field ofsearch queries on a computer network.

2. Description of the Related Art

The Internet is a global system of interconnected computer networks thatprovide access to a vast array of information. The World Wide Web (WWW)is an information sharing model built on top of the Internet, in which asystem of interlinked hypertext documents are accessed using particularprotocols (i.e., the Hypertext Transfer Protocol and its variants).

Because of the enormous volume of information available via the WWW andthe Internet, and because the available information is distributedacross an enormous number of independently owned and operated networksand servers, locating desired content on the WWW and the Internetpresents challenges.

Search engines have been developed to aid users in locating desiredcontent on the Internet. A search engine is a computer program thatreceives a search query from a user (e.g., in the form of a set ofkeywords) indicative of content desired by the user, and returnsinformation and/or hyperlinks to information that the search enginedetermines to be relevant to the user's search query.

Search engines typically work by retrieving a large number of WWW webpages and/or other content using a computer program called a web crawlerthat browses the WWW in an automated fashion (e.g., following everyhyperlink that it comes across in each web page that it browses). Theretrieved web pages and/or content are analyzed and information aboutthe web pages or content is stored in an index. When a user issues asearch query to the search engine, the search engine uses the index toidentify the web pages and/or content that it determines to best matchthe user's search query and returns a list of results with thebest-matching web pages and/or content. Frequently, this list is in theform of one or more web pages that include a set of hyperlinks to theweb pages and/or content determined to best match the user's query.

There are at least two general types of search engines accessible viathe Internet: general-purpose search engines and site-specific searchengines. As used herein, the term “general-purpose search engine” meansa search engine that provides search results that include web pagesand/or content (or hyperlinks to web pages and/or content) hosted on atleast two different and independent web sites or domains. Generalpurpose search to engines attempt to index and provide search resultsfrom content distributed across a wide swath of the Internet. Examplesof general purpose search engines include Google™, operated by Google,Inc. of Mountain View, Calif.; Yahoo!™, operated by Yahoo!, Inc. ofSunnyvale, Calif.; and Bing™, operated by Microsoft Corp. of Redmond,Wash.

As used herein, the term “site-specific search engine” means a searchengine that provides search results that include web pages and/orcontent (or hyperlinks to web pages and/or content) hosted on only oneweb site or domain. Site-specific search engines are frequently used byoperators of web sites to allow users to find specific web pages orcontent on their web sites. For example, the web site of an onlineretailer (or “e-tailer”) may include a site-specific search engine thatfacilitates a user locating web pages for products sold by the retailer.

SUMMARY

Some embodiments of the invention provide a method, performed by atleast one computer, comprising acts of: (A) receiving a query from adevice and location data indicating a location of the device, thelocation data having a level of specificity; (B) in response to thequery being received, identifying at least one first search engine towhich to submit a representation of the query; (C) determining whetherthe level of specificity of the location data received in (A) issufficient for the at least one first search engine; (D) if the level ofspecificity of the location data is sufficient, instructing the deviceto issue the representation of the query to the at least one firstsearch engine; and (E) if the level of specificity of the location datais not sufficient, instructing the device to send, to the at least onecomputer, location data at a greater level of specificity.

Some embodiments of the invention provide at least one computer readablemedium having encoded thereon instructions which, when executed by atleast one computer, perform a method comprising acts of: (A) receiving aquery from a device and location data indicating a location of thedevice, the location data having a level of specificity; (B) in responseto the query being received, identifying at least one first searchengine to which to submit a representation of the query; (C) determiningwhether the level of specificity of the location data received in (A) issufficient for the at least one to first search engine; (D) if the levelof specificity of the location data is sufficient, instructing thedevice to issue the representation of the query to the at least onefirst search engine; and (E) if the level of specificity of the locationdata is not sufficient, instructing the device to send, to the at leastone computer, location data at a greater level of specificity.

Some embodiments of the invention provide at least one computer,comprising: at least one processor programmed to: receive a query from adevice and location data indicating a location of the device, thelocation data having a level of specificity; in response to the querybeing received, identify at least one first search engine to which tosubmit a representation of the query; determine whether the level ofspecificity of the received location data is sufficient for the at leastone first search engine; if the level of specificity of the locationdata is sufficient, instruct the device to issue the representation ofthe query to the at least one first search engine; and if the level ofspecificity of the location data is not sufficient, instruct the deviceto send, to the at least one computer, location data at a greater levelof specificity.

BRIEF DESCRIPTION OF DRAWINGS

In the drawings:

FIG. 1 is a Venn diagram showing sets of search results from multipledifferent search engines;

FIG. 2 is a flow chart of an illustrative process for sending searchqueries to and receiving search results from multiple search engines, inaccordance with some embodiments;

FIG. 3 is a block diagram of a computer environment in which someembodiments may be implemented;

FIG. 4 is a block diagram of a client device that executes anapplication program for querying multiple search engines and anautomated speech recognizer for performing speech recognition on voicesearch queries, in accordance with some embodiments;

FIG. 5 is a block diagram of a client device that executes anapplication program for querying multiple search engines and a serverthat executes an automated speech recognizer and provides speechrecognition services for the client device to perform speech recognitionon voice search queries, in accordance with some embodiments;

FIG. 6 is a block diagram of a computing environment in which someembodiments may be implemented;

FIG. 7 is a block diagram in which audio data of a voice speech query isrecognized using multiple different language models, in accordance withsome embodiments;

FIG. 8 is a block diagram of a server that instructs a client devicewhich search engines to query in response to receiving a user-suppliedsearch query, in accordance with some embodiments;

FIG. 9 is a diagram showing Universal Resource Locators (URLs) that maybe generated to query multiple search engines, in accordance with someembodiments;

FIG. 10A is a depiction of a display of a client device in which searchresults from multiple search engines may be displayed;

FIG. 10B is a depiction of a display of a client device in which searchresults from multiple search engines may be displayed;

FIG. 11 is a block diagram of an illustrative computing device on whichaspects described below may be implemented;

FIG. 12 is a flow chart of an illustrative process for identifying thetype of information and/or action a user seeks; and

FIG. 13 is a flowchart of an illustrative process for determiningwhether sufficiently accurate location data is available for use.

DETAILED DESCRIPTION

The inventors have recognized that a given user-specified search querymay to indicate an interest in searching for any of several differenttypes of information pertaining to the subject of the query. Forexample, for the search query “Miles Davis,” the user may be interestedin obtaining biographical information about Miles Davis, listening tosamples of or purchasing Miles Davis' music, and/or social networkingwith others who are interested in Miles Davis. Historically, to obtainthese different types of information, a user may have had to enter thesearch string “Miles Davis” into three different search engines. Forexample, the user might have entered this search string into the searchengine for an encyclopedia web site to obtain biographical information,into the search engine for a web site that sells music to listen to orpurchase music, and into the search engine of a social networking siteto connect with others. The inventors have recognized that this processis often time consuming and laborious, as the user must navigate tomultiple different web sites (and potentially manage multiple differentbrowser windows or tabs) and enter the same search string repeatedly.

The inventors have also recognized that, because different searchengines index web pages and/or content in different ways, indexdifferent universes of content, and/or use different algorithms todetermine which web pages and/or content best match a particular searchquery, different search engines may provide different (though possiblyoverlapping) sets of search results in response to the same searchquery. This concept is illustrated by the Venn diagram in FIG. 1. Eachset depicted in the Venn diagram of FIG. 1 represents the search resultsfrom one of four hypothetical search engines (i.e., search engine one,search engine two, search engine three, and search engine four) providedin response to a user-provided search query for the keywords “MilesDavis.” In FIG. 1, set 101 includes results provided from search engineone and includes search results “Q,” “R,” “S,” and “T.” Set 103 includesresults provided from search engine two and includes results “R,” “S,”and “Y.” Set 105 includes results provided from search engine three andincludes results “S,” “T,” and “X.” Set 107 includes results providedfrom search engine four and includes result “Z.” As shown in FIG. 1,some search results are included only in one of the sets and, as such,are returned from only one of the four search engines, while othersearch results are included in multiple of the sets and, as such, arereturned from two or more of the search engines.

When a search engine returns a set of search results in response to asearch query, the search engine generally returns the results in anordered list. The list may be ordered by relevance, may be ordered basedon money paid for higher positions in the search results, and/or may beordered based on other criteria. For example, web pages or content thata search engine has determined to be most relevant may be at the top ofthe list of the results, while web pages or content that the searchengine has determined to be less relevant may be farther down in thelist of results. As another example, a particular electronicsmanufacture may pay an electronics e-tailer to list its televisionshigher up in the list when users enter search queries including the word“television” into the site-specific search engine for the e-tailer's website.

The inventors have appreciated that, because different search enginesuse different algorithms to determine in what order web pages and/orcontent are to be listed in the search results, even in a situation inwhich two different search engines include the same web page or piece ofcontent in their search results in response to a particular search query(e.g., “Miles Davis”), any particular web page or piece of content maybe at or near the top of the list of results provided by a first of thetwo search engines (e.g., because the first search engine has determinedthe web page or piece of content to be particularly relevant in theuniverse of content in evaluates), but may be further down the list ofresults provided by a second of the two search engines (e.g., becausethe second of the two search engines has determined the web page orpiece of content to be less relevant in the universe of content itevaluates). Similarly, some results that are listed near the top of thelist of search results from the second search engine may be listed muchlower in the list of results provided by the first engine. The inventorshave recognized that users are more likely to notice and access resultsthat are near the top of the list. Thus, by using only one of the twosearch engines, a user may not notice or access results that may behighly relevant to the user.

As such, the inventors have recognized that issuing a user-specifiedsearch query to only a single search engine increases the chances ofsome web pages and/or pieces of content that are potentially relevantnot being included the search results or being listed low enough in thereturned list of search results that the user disregards them. Inaddition, the inventors have recognized that issuing such a search queryto only a single search engine may limit the content that is returned toonly the type of content that is to search and/or indexed by that searchengine, while the user may be interested in other, different types ofcontent pertaining to that search query.

Some conventional web-based software programs, referred to as metasearchengines or search engine aggregators, receive a user-specified searchquery, issue the search query to multiple search engines, receiveresults from each of these search engines, remove duplicates, aggregatethese search results into a single list, and display this list ofaggregated search results to the user. One example of such a metasearchengine is Dogpile™, operated by Infospace, Inc. of Bellevue, Wash.However, the inventors have recognized that these metasearch engineshave a number of disadvantages.

First, the search engines that are queried by these metasearch enginesinclude only general-purpose search engines, and do not include anysite-specific search engines.

Second, because metasearch engines, in response to a user-specifiedsearch query, display to the user a single list of aggregated searchresults from a number of different search engines, the user has noability to see which search engine provided which search result, to seehow relevant each of the plurality of search engines deemed a particularsearch result to be, or to look at the full set of results returned fromany one particular search engine.

Third, metasearch engines run on one or more servers that receive auser-specified search query issued from a browser or other softwareapplication executing on the user's client device and issue the searchquery from the server(s) to the plurality of search engines that are tobe queried. Because a metasearch engine may receive search queries fromand provide aggregated search results to a large number of users, theserver(s) on which the metasearch engine operates may regularly issue avery large number of search queries to the search engines whose resultsit aggregates. One consequence of this is that a search engine mayreceive hundreds of thousands or even millions of search queries dailyfrom the same server or IP address, and may perceive this behavior as asingle user issuing an enormous number of search queries to the searchengine. Many search engine operators would consider this behavior to bean abuse of the search engine service and/or a potential denial ofservice attack and would take action to block search queries to theirsearch engines from an IP address issuing such a large to number ofqueries. For this reason, a metasearch engine operator must typicallyobtain contractual agreements with the search engine operators whosesearch engine results are aggregated by the metasearch engine to allowthe metasearch engine to issue a large number of search queries to thesesearch engines.

Fourth, these metasearch engines do not provide any capability for theuser to control to which search engines his or her search query is to beprovided. Rather, existing metasearch engines have a fixed set of searchengines to which every search query is provided. Thus, the user has nocontrol over which search engines' results are provided in response to asearch query and the metasearch engines do not perform any customizationof which search engines are queries based on which user issued thesearch query or based on the content of the user's search query.

While some embodiments of the invention address above-discusseddeficiencies of existing metasearch engines, it should be appreciatedthat not every embodiment addresses all of the deficiencies discussedabove, and some embodiments may not address any of these deficiencies.As such, it should be understood that the invention is not limited toembodiments that address all or any of the above-described deficienciesof metasearch engines.

Some embodiments described below are directed to techniques forreceiving a user-specified query comprising content and, in response toreceiving the query, identifying, based at least in part on the contentof the query, at least one search engine to which a representation ofthe query is to be submitted. For example, some embodiments of theinvention are directed to determining, upon receiving a user-specifiedsearch query, the type of information which the user seeks, andcustomizing the search engines to which the query is submitted. Thisdetermination may, for example, be based on the content of the user'squery, the location of the user when the search query was received, theuser's browsing and/or query history, the browsing and/or query historyof other users, any combination of the foregoing, and/or otherinformation. Upon determining the type of information the user seeks, arepresentation of the query may be submitted to multiple search enginesthat are selected because they are known to provide access toinformation of that type. Thus, the search engines to which queries aresent and for which results are displayed may not be static in someembodiments, but rather may be to dynamically determined, so that thesearch engines may vary from query to query for a particular user and/orbetween different users.

As a simple example, given a user's query “Tom Brady,” embodiments ofthe invention may determine (e.g., based on the content of the query,the user's previous search and/or browsing history, etc.) that the userlikely seeks information relating to sports. As such, embodiments of theinvention may cause a representation of the user's query to be submittedto one or more search engines provided by sports-related websites, andcause results generated by the search engine(s) to be presented to theuser.

Some embodiments of the invention are not limited to identifying only asingle type of information as that which the user seeks, and maydetermine two or more types. Using the example query “Tom Brady” toillustrate, embodiments of the invention may also acknowledge thepossibility that the user seeks celebrity-related information (e.g.,based on the query's content, the user's previous search and/or browsinghistory, etc.). As a result, some embodiments of the invention may causea representation of the user's query to be submitted to one or moresearch engines provided by celebrity-related websites, and cause resultsgenerated by the search engine(s) to be presented to the user, insteadof or in addition to the results generated by sports-related websites.

In seeking to determine the intent of the user, the possibility existsthat the type of information sought by the user may be incorrectlyidentified. To address this risk, some embodiments of the invention maysubmit a representation of the user's query to search engines providingaccess to other types of information than that which was determined thatthe user's query may be directed to. For example, a representation ofthe user's query may be submitted to one or more general-purpose searchengines, and results generated by the search engine(s) may be presentedto the user, in addition to the information generated by the morespecific search engines (e.g., sports-related and/or celebrity-relatedwebsites in the Tom Brady example) which provide information determinedto be that in which the user is likely interested. In this way, thesystem provides the benefit of focused and highly relevant results whenit determines the user's intent correctly, while having the “safety net”of providing at least some relevant information (e.g., via thegeneral-purpose search engine) if it determines the user's intentincorrectly.

While some embodiments that attempt to determine the user's intentprovide the safety net or fallback option of returning other resultsreturned by additional search engines, not all embodiments are limitedin this respect. For example, some embodiments of the invention may onlycause results generated by more specific search engines to be presented.

In some embodiments, results generated by the search engines may bearranged based on an assessment of how likely it is that each searchengine provides the type of information the user seeks. For example,referring to the Tom Brady example, if it is determined that the usermost likely seeks sports-related information, then results generated byone or more search engines provided by sports-related website(s) may bedisplayed most prominently. If it is determined to also be possible thatthe user seeks celebrity-related information, then results generated byone or more search engines provided by celebrity-related website(s) maybe displayed less prominently than that which is provided by thesports-related websites. If the user's query is also submitted togeneral-purpose search engines (e.g., as a “backup option”), then theresults generated by those search engines may be displayed leastprominently. It should be appreciated, however, that an arrangement ofresults generated by various search engines in this manner is optional,and that embodiments of the invention are not limited to displayingresults in a manner that is influenced by an assessment of how likely auser seeks a particular type of information.

Displaying some results more prominently than others may be accomplishedin any of numerous ways, some non-limiting examples of which aredescribed below. In one example, results from multiple search enginesmay be displayed concurrently on a display which employs a carouselmetaphor, in which each search engine from which results were receivedoccupies a slot. Examples of a carousel are shown in FIGS. 10A-10B, anda commercialized example is the Dragon Search mobile phone applicationmade available by the Assignee of this application, NuanceCommunications, Inc. The user may select a particular slot to viewresults generated by the corresponding search engine. Using the exampleabove to illustrate, the slots corresponding to search engines providedby sports-related websites may be displayed most prominently by, forexample, placing the corresponding slots in the center of the carousel,toward the left, or in any to other suitable position. The slotscorresponding to the search engines provided by celebrity-relatedwebsites may, for example, be placed in slots that are deemed lessprominent than those corresponding to the sports-related websites,followed by the slots corresponding to general-purpose search engines.In some embodiments, the less prominent slots may be visible when theresults are initially presented, whereas in other embodiments they maynot, so that the user may need to take an action (e.g., scroll) to viewthe results for those slots.

As should be appreciated from the foregoing, some embodiments of theinvention determine, upon receiving a user's query, the type ofinformation the user most likely seeks, and cause information of thattype to be presented most prominently to the user, and may cause othertypes of information (e.g., determined to be less likely than theidentified type but still possibly the type the user seeks) to bepresented as “backup.” As a result, some embodiments of the inventionmay be aggressive in presenting specialized information of the typeidentified to be most likely that which the user seeks, while alsoallowing for the possibility that the user seeks information that is notlimited to any particular type.

FIG. 2 depicts an example process 200 that may be performed todetermine, upon receiving a user's query, the type of information thatthe user likely seeks, and to present that type of information (e.g.,alone or among one or more other types of information) to the user.Process 200 may, for example, be performed by an application programexecuting on a client device 250 employed by the user and/or a computer275 that is external to the client device (e.g., in communication withclient device 250 via a communication medium, not shown). Client device250 may be any suitable type of computing device having hardware capableof executing an application program. Examples of client devices that maybe used include a laptop or desktop personal computer, a personaldigital assistant (PDA), a mobile phone (e.g., a smartphone), a servercomputer, and/or various other types of computing devices. Externalcomputer 275 may also include any suitable type of computing device,such as a laptop or desktop personal computer, server computer, and/orother type of computing devices. Collectively and/or individually, theclient device and external computer may provide speech recognitioncapability, so that a user-specified query may be submitted via voiceinput. However, embodiments of the invention are not limited to systemshaving speech recognition capability, and are not limited to systemscomprising mobile telephones or smartphones.

Process 200 begins in act 201, wherein an application program on clientdevice 250 receives a user-specified query (e.g., via voice and/or textinput). As discussed in greater detail below, the query may be receivedin any of numerous ways via any of numerous user interfaces. In theexample process shown in FIG. 2, the application program executing onthe client device causes a representation of the user-specified query tobe transmitted to external computer 275 in act 203, which receives it inact 205 and identifies the type of information and/or action which theuser seeks in act 207. As described in detail below, the representationof the query can take any of numerous forms, as can the identificationof the information and/or action specified by the query. Externalcomputer 275 generates information (e.g., comprising search queries tobe submitted to the identified search engines) in act 209, and transmitsthe information to the application program executing on the clientdevice in act 211. After receiving the transmitted information in act213, the application program executing on client device 250 processesthe information (e.g., by issuing search queries to each identifiedsearch engine) in act 215. If act 215 includes issuing search queries tosearch engines, then the application program executing on client device250 receives results generated from those queries in act 217. Resultsare displayed to the user in act 219. Example process 200 thencompletes.

It should be appreciated that example process 200 represents only onemanner of implementing embodiments of the invention. For example, notall embodiments of the invention are limited to employing a clientdevice executing an application program and an external computer, asnumerous other system configurations may alternatively be employed. Forexample, some embodiments of the invention may receive a user-specifiedquery, identify search engines making available the types of informationthe user may seek, generate search queries and display resultstherefrom, all on a single device. In another example, some embodimentsof the invention may employ multiple computers external to the clientdevice to identify search engines making available the types ofinformation the user may seek and generate search queries. Variations onexample process 200 may be performed by any of numerous componentconfigurations, as embodiments of the invention are not limited in thisrespect.

Further, it should be appreciated that numerous variations on exampleprocess 200 are possible. For example, some variations may include oneor more of acts 201-219 being performed in a different sequence thanthat which is described herein, may omit one or more of acts 201-219,and/or may include acts not described. Embodiments of the invention arenot limited to being implemented in any particular manner.

The acts comprising example process 200 are described in detail below.

I. Receiving a user-specified search query

In act 201, an application program executing on client device 250receives user input specifying a search query. A search query may takeany suitable form, and may not be bound by a predefined syntax. Forexample, a search query may be free-form, and comprise any informationsusceptible to representation via a string of one or more characters.For example, a search query may comprise a name of a person (e.g., “TomBrady”), place (e.g., “Saratoga Springs”), thing (e.g., “New York StockExchange”), any combination thereof (e.g., “ski shops in Park City”),and/or otherwise convey a user's desire to search for information on oneor more particular topics or concepts.

User input defining a search query may be received in any of a varietyof ways and in any of a variety of formats. For example, in someembodiments, the user input may be received as text and may be receivedvia an input device (e.g., a keyboard, touch screen, mouse, and/or anyother input device) integrated into or coupled to the client device. Insome embodiments, user input may be received as audio via a microphonethat is integrated into or coupled to the client device. For example, auser may speak a search query into the microphone and the applicationprogram may receive the user's voice input as audio data. In someembodiments, the client device may provide a number of different optionsfrom which the user may select to provide a search query. For example,in some embodiments, the client device may permit the user to input thequery either in voice format (e.g., by speaking the query into amicrophone) or in text format (e.g., by typing the query on a keyboardor a touchscreen).

In the example process 200, a representation of the user-specified queryreceived in act 201 is transmitted to external computer 275 in act 203,and received by the external computer in act 205. In embodiments whereinthe query received in act 201 is specified via voice input, automatedspeech recognition may be performed on audio data representing the voiceinput to obtain a recognition result (e.g., in textual form) that may beused to identify one or more search engines making available the type(s)of information the user seeks, and to generate queries to be submittedto those search engines. However, not all embodiments of the inventionare limited to performing automated speech recognition. For example, oneor more of the search engines to be queried may have a speech interface,so that audio data may be provided to such search engines instead ofproviding a text search query based on automated speech recognition ofthe audio. Embodiments of the invention are not limited to anyparticular implementation.

If performed, automated speech recognition may be conducted by clientdevice 250, external computer 275, or a combination thereof. Automatedspeech recognition techniques are known, and any suitable automatedspeech recognition technique may be used to obtain a recognition resultfor the audio data.

As shown in FIG. 4, in some embodiments, the client device may executean automated speech recognizer, which is a software program thatperforms automated speech recognition on audio data, and the applicationprogram may use the automated speech recognizer to obtain a recognitionresult of the audio data. In FIG. 4, client device 401 executes theapplication program 403 that performs at least some of theabove-described acts of process 200 (e.g., those performed by the clientdevice) and an automated speech recognizer 405. When application program403 receives user-supplied audio data 407, it may pass the audio data toautomated speech recognizer 405. Automated speech recognizer 405 mayperform automated speech recognition on the audio data to obtain arecognition result 409 and may return recognition result 409 toapplication program 403 for use in formatting queries for the speechengines.

The inventors have recognized that some client devices may not havesufficient computing resources to execute an automated speech recognizerthat operates with a desired level of accuracy and/or efficiency. Thatis, for example, a client device may not have sufficient memory to storeacoustic models, language models, grammars, and/or other components ofsuch a speech recognizer and/or may not have a processor powerful enoughto perform automated speech recognition at a desired level ofefficiency. This may occur in any number of situations, such as when theclient device is a handheld device, such as a PDA or mobile phone.

Thus, in some embodiments, automated speech recognition on audio datathat originates from a user may be performed by the external computer.As shown in FIG. 5, client device 501 executes application program 503that receives voice input supplied by a user in the form of audio data.Automated speech recognition may be performed on a representation of theaudio data by an automated speech recognizer 507 executing on a server505. That is, automated speech recognition may be performed on the fullaudio data, or some processed representation of it. Thus, as shown inFIG. 5, in some embodiments, application program may send theuser-supplied audio data 509 to automated speech recognizer 507executing on server 505. The audio data 509 may be sent to the automatedspeech recognizer in any of a variety of possible ways, and may beprocessed on the client device and/or on the server before being passedto the speech recognizer. For example, in some embodiments, theuser-supplied audio may be captured by a microphone andanalog-to-digital conversion may be performed (on the client or server)on the captured microphone audio to generate digital audio data. Thisdigital audio data may be supplied to automated speech recognizer 507.In some embodiments, some processing of the digital audio data may beperformed on the client and/or server prior to sending this data to theautomated speech recognizer. For example, the digital data may becompressed or some pre-processing may be performed to extract parametersof the audio that are useful in automated speech recognition. Theextracted parameters may be passed to automated speech recognizer 507 asthe representation of the audio data or may be sent along with otheraudio data. In embodiments that use compression, any of a variety ofpossible compression algorithms may be used. For example, in someembodiments, the Speex™ codec may be used.

Thus, it should be appreciated that, as used herein, the term “audiodata” may refer to an analog audio signal, a digital representation ofan analog audio signal, a compressed representation of such a digital oranalog signal, a set of features or parameters extracted from thedigital or analog signal, any combination of the foregoing, and/or otherinformation.

Automated speech recognizer 507 may perform automated speech recognitionon audio data 509 and return to application program 503 a response 511that comprises results from audio data 509. The results may take anysuitable form. As one non-limiting example, the results may comprise atext recognition result obtained by automated speech recognizer 507 fromperforming automated speech recognition on the audio data.

In the example of FIG. 5, client device 501 is depicted as communicatingdirectly with server 505. It should be appreciated that this depictionis provided merely to facilitate an understanding of the types ofinformation sent between client device 501 and server 505, and thatclient device 501 may communicate with server 505 in any of a variety ofways and via any suitable communication medium, including, for example,via the Internet.

In the example of FIG. 4, automatic speech recognition on audio datareceived at a client device is performed on the client device, and inthe example of FIG. 5, audio data is provided from the client device toa server and the server performs automatic speech recognition. However,in some embodiments, automatic speech recognition may be performedpartially by the client device and partially by the server. For example,a portion of an ASR process may be performed on the client and a portionmay be performed on the server. This may be done, for example, to enablethe client device to perform a portion of the speech recognitionprocess, but leave computationally intensive portions for the server.

In some embodiments, one or more servers (like server 505) may provideautomated speech recognition services for multiple client devices. FIG.6 shows a computing environment in which each of a plurality of users601 a, 601 b, . . . 601 n operates a client device 603 a, 603 b, . . . ,603 n. Each client device 603 may execute an application program thatsends search queries via communication medium 607 to search engines 609a, 609 b, 609 c, . . . , 609 n, and displays the results of these searchqueries to its respective users (e.g., by executing some of the acts ofprocess 200 of FIG. 2). Communication medium 607 may comprise anysuitable communications infrastructure, employing any suitablecommunication protocol(s), including wired and/or wirelesscommunications protocols. For example, communication medium 607 maycomprise one or more intranets, local area networks (LANs), wide areanetwork (WANs), the Internet, or any combination thereof.

Server 605 may provide automated speech recognition services to supportthese searches when the user enters a query audibly. Thus, inembodiments in which client devices 603 receive user-supplied searchqueries in the form of audio data, the client devices 603 may send someor all of the audio data via communication medium 607 to server 605 tohave automated speech recognition performed on the audio data, and mayreceive a recognition result and/or other information (e.g., searchqueries and an identification of search engines to issue them to) viacommunication medium 607 based on the recognition result in response.

In some alternate embodiments, server 605 may provide otherfunctionality (either instead of or in addition to ASR services) tofacilitate issuance of search queries from a client device 603 to one ormore of search engines 609. Some examples of this additionalfunctionality are described in detail below. For the sake of simplicity,in the example of FIG. 6, only one server 605 is depicted. However, itshould be understood that, as is conventional for Internet servers thatreceive a high volume of traffic, any number of servers may be used toprovide the functionality of server 605 that is described herein and thetraffic load from client devices 603 may be balanced across theseservers using any of a variety of load balancing techniques. Inaddition, the functionality of the servers may be distributed in anysuitable way, so that each server need not perform the same functionsand may be arranged to share the functions performed by the set ofservers collectively.

In embodiments in which a user supplies a search query by voice (i.e.,in the form of audio data), a number of techniques may be used tofacilitate automated speech recognition of the audio data to generatesearch queries. These techniques may be employed in embodiments in whichautomated speech recognition is performed entirely on the client device,in embodiments in which automated speech recognition is performedentirely on a computer external to the client device, and in embodimentswherein ASR is distributed between the client device and an externalcomputer.

In some embodiments, one or more language models used by the automatedspeech recognizer in performing speech recognition on the audio data ofthe user-supplied search query may be selected based on the searchengine(s) which are to be queried. For example, as shown in FIG. 7, insome embodiments, the automated to speech recognizer may performrecognition on the same audio data using multiple different languagemodels to yield multiple different recognition results. In FIG. 7,speech recognition is performed on audio data 701 using language models703 a, 703 b, . . . , 703 n to generate recognition results 705 a, 705b, . . . , 705 n. Depending on the content of audio data 701 and thedifferences between language models 703, each of recognition results 705may be the same, each may be different from the others, or some may bethe same and some may be different. One or more of language models 703may be associated with a particular search engine and the recognitionresult generated using a language model that is associated with aparticular search engine may serve as the basis for the search querythat is issued to that search engine.

Thus, for example, in some embodiments, a language model that has beentrained on the content that is indexed by a particular search engine maybe used to generate the recognition result that serves as the basis forthe query that is issued to that search engine. This results in a querythat is more likely to yield relevant search results for that searchengine.

This technique may be particularly useful when issuing queries tosite-specific search engines that index a limited amount of content thatis generally directed to a particular subject area. For example, anonline shoe retailer may have a site-specific search engine for itse-commerce web site. As such, the content of this web site may includeshoe-specific terminology and phrases, such as the name of shoe brandsand models, descriptions of shoe properties, and various othershoe-specific information. By training the language model used by aspeech recognizer on the content of this web site, the recognizer ismore likely to accurately recognize speech that includes theshoe-specific terminology and phrases.

Thus, for example, if a user-supplied search query in the form of audioinput is received and is to be issued to both a site-specific searchengine and a general-purpose search engine, the audio input may berecognized using a first language model (e.g., a language model that hasbeen trained on content related to the content indexed by thesite-specific search engine) to generate a recognition result thatserves as the basis for the query to be issued to the site-specificsearch engine and may also be recognized using a second language model(e.g., that has been trained on more general content), different to fromthe first language model, to generate a recognition result that servesas the basis for the query to be issued to the general-purpose searchengine.

In the example described above, the recognition result that is obtainedby performing speech recognition using a language model that isassociated with a particular search engine serves as the basis for thesearch query that is issued to that search engine. However, in someembodiments, rather than issuing recognition results obtained usingdifferent language models to different search engines, multiplerecognition results may be obtained using different language models andthe recognition result with the highest score or confidence value mayserve as the basis of a search query that is issued to multiple searchengines. For example, in FIG. 7, each of recognition results 705 may begenerated by performing automatic speech recognition on audio data 701using a corresponding language model. In some embodiments, one ofrecognition results 705 may be selected to be the result that serves asthe basis for generating a search query to be issued to multiple searchengines. This selection may be made in any suitable way. For example, asdiscussed above, the recognition result that has the highest score orconfidence value as a result of the speech recognition may be selected,or any other suitable criterion or combination of criteria may be used.

In situations in which speech recognition is performed on the same audiodata using multiple language models, performing multiple separate andindependent speech recognition processes with each of the differentlanguage models may increase the latency in obtaining recognitionresults and/or may increase the amount of processing resources used inperforming speech recognition.

Thus, in some embodiments, rather than performing an entire independentspeech recognition process on the audio data for each language model,one generalized language model may be used to create a lattice ofhypothesized word sequences, and this lattice may be re-scored using oneor more selected language models to yield recognition result(s)particularly for the selected language model(s). In this way, the timeand processing resources consumed in generating the recognition resultsmay be reduced because the same lattice is used in producing eachrecognition result and thus only needs to be generated once.

In the example of FIG. 7, audio data 701 is shown as being provideddirectly to each of language models 703. This depiction is providedmerely to facilitate an understanding of how multiple language modelsmay be used, and it should be understood that, in practice, the languagemodels may not operate directly on audio data. For example, a latticewith multiple hypothesized sequences of words may be obtained from audiodata using an acoustic model and a general language model, andspecialized language models 703 may be used to rescore the generallattice and obtain a sequence of words appropriate to a specific searchengine.

In some embodiments, rather than using multiple different languagemodels in generating search queries to be provided to search engines,one language model may be selected from a plurality of availablelanguage models. For example, in some embodiments, one language modelmay be identified as being well-suited for recognizing a particularvoice query, and that language model may be used in recognizing thequery.

Any of a variety of possible criteria or combinations of criteria may beused to select a language model to be used in recognizing a particularvoice query. For example, in some embodiments, the content of the querymay be used as a basis for selecting a language model to be used. Forexample, a two-pass process may be used whereby in a first passautomatic speech recognition of a voice query or a portion of a voicequery is performed using a general language model that doesn't have aspecialized vocabulary. Such a language model may recognize some of thewords in the query, but may not accurately recognize all of the words.The recognition result obtained from the first-pass may be used toclassify the query based on topic or category. Any suitableclassification technique may be used, and a variety are known.

Based on the classification, a language model that has a specializedvocabulary directed to the identified topic or category may be selected.After selection of the language model, a second-pass of automatic speechrecognition may be performed using the selected language model. Forexample, if after the first-pass it is determined that the voice queryincludes word such as “directions,” “street,” or “road,” it may bedetermined that voice query is map-related query, and a map-relatedlanguage model (e.g., a language model that includes city, street, andplace names for the United States and/or to other countries) may beselected and used in the second-pass.

As discussed in greater detail below, the recognition result from thesecond-pass may serve as the basis for one or more search queries to beprovided to one or more search engines. In addition, as discussed inmore detail below, the search engine or engines that are queried may beselected based on the language model that was selected to recognize thequery in the second-pass, and/or based on the content of the recognitionresult obtained from the second-pass.

In some embodiments, a language model that is used to recognize a voicesearch query supplied from a user as audio data may be updated based ontopics of interest. This may be done in any of a variety of possibleways. Current events frequently result in search engines receiving alarge number of search queries that include keywords associated with aparticular event, including keywords that may have been previouslyunused and not the subject of frequent searches. For example, if apopular new movie is playing in movie theaters, search engines mayreceive a large number of search queries from different users thatinclude the name of the movie, the names of the actors in the movie, orother keywords associated with the movie. Similarly, if a popular orhigh-profile sporting event is upcoming, search engines may receive alarge number of queries that include the names of the teams and/orathletes participating in the event, or an event in popular culture maythrust a person previously unknown to the general public into thelimelight (e.g., a person winning the lottery, a person accused of ahigh-profile crime, a person involved in a high profile scandal, etc.)

In some embodiments, words and/or phrases associated with topics ofrecent user interest may be identified, and the language model may beupdated to improve the accuracy of recognizing these words and/orphrases. These words and/or phrases may be identified in any of avariety of ways. For example, in some embodiments in which user-suppliedqueries (e.g., in audio format, text format, or some other format) areprovided from a plurality of client devices to a server or set ofservers, the server or set of servers may identify frequently usedsearch terms or phrases in these search queries and the language modelmay be updated based on these frequently used search terms. In otherembodiments, words and/or phrases associated with topics of interest maybe identified to manually by one or more humans tasked with identifyingsuch words or phrases.

One or more language models may be updated in this manner with anydesired degree of frequency. For example, in some embodiments, alanguage model may be updated at a regular interval, such as once perday, once every two days, twice per day, once per week, or any of avariety of other possible intervals. In some embodiments, a languagemodel may be updated in response to a user action, such as an indicationfrom the user to update the language model or in any other suitable way.

In some embodiments, words provided in user search queries, wordsobtained from web crawling, and/or words from other sources may becollected and used to update one or more language models. A word countindicating the number of occurrences of the words in the source data maybe maintained and words whose word count exceeds a threshold value maybe added to the language model(s).

In some embodiments, an auto-pronunciation (e.g., a sequence ofphonemes) for each word to be added to the language model may begenerated and included in the language model(s) with its correspondingword. In some embodiments, a human reviewer (e.g., a linguist) mayreview and, if warranted, revise the automatically generatedpronunciation before it is added to the language model(s).

In some embodiments, the language model(s) used in automated speechrecognition of search queries may be updated using feedback indicatingwhich search results a user selected. This may be done in any of avariety of possible ways. For example, in some embodiments, an automatedspeech recognizer may generate a recognition result from audio data of auser-supplied voice search query. This recognition result may serve asthe basis for one or more search queries that are provided to one ormore search engines, and search results from the one or more searchengines may be returned and displayed to the user in response to thequery or queries. The user may select one or more of the search resultsto view a particular web page or piece of content. In some embodiments,information about which search result(s) the user selected may be usedto update the language model(s) used by the automated speech recognizer.For example, a user may speak the search query “Willie Mays” and theautomated speech recognizer may incorrectly recognize the speech as“Willy Maze.” As such, a search query for the phrase “Willy Maze” may beissued to one or more search to engines. However, one or more of thesearch engines may return in its list of results a hyperlink to a webpage with biographical information about Willie Mays. If the userselects this hyperlink to access that web page, the content of the webpage may be used to update the language model(s). This may increase thelikelihood that the next time a user speaks the phrase “Willie Mays,” itwill be correctly recognized.

In some embodiments, the language model(s) used for a particular usermay be customized to that user's speech, using previous utterancesobtained from that user. This may be done in any of a variety of ways.For example, in some embodiments, voice data received from a particularspeaker may be represented as a bag-of-word feature vector, with thegoal of discovering a set of representative latent speakers to “explain”the data. The outcome is that, for example, similar speakers areclustered together in a probabilistic fashion. These clusters may beused to build a set of latent speaker language model components. Usingthese latent components, adaptation for a particular user is performedvia estimating the speaker-specific linear interpolation weights of thelanguage model components using the received voice data from theparticular speaker for which adaptation is being performed.

In some embodiments, the language model(s) used for a particular usermay be updated based on the user's historical browsing information. Thismay be accomplished in any of a variety of ways. For example, based onthe user's browsing history, web sites that are frequently visited maybe identified. The language model(s) used to recognize speech queriesmay be re-trained using a training corpus in which content from thefrequently visited web sites is emphasized.

Any or all of the above-described techniques for selecting a languagemodel to facilitate automated speech recognition can be used incombination with each other. In addition, it should be appreciated thatthese techniques for selecting a language model need not be used, and insome embodiments the same language model may be used to process allvoice input.

II. Determining the Type of Information and/or Action the User Seeks

In act 207 of example process 200, external computer 275 identifies thetype(s) of information that the user seeks and/or action(s) that theuser wishes to perform. This may be performed in any of numerous ways.An example process 1200 for determining, upon receiving a user-specifiedquery, the type(s) of information that the user seeks and/or theactions(s) that the user wishes to perform is depicted in FIG. 12. Theexample process 1200 is but one technique which may be used to determineone or more types of information sought by the user and/or actions theuser wishes to perform. Numerous other techniques for doing so arepossible.

At the start of process 1200, automated speech recognition is performedon audio data defining the query in act 1205. As noted above, automatedspeech recognition may be performed in any of numerous ways, asembodiments of the invention are not limited to employing any particulartechnique(s). In some embodiments, the output of act 1205 is a textualrepresentation of the user-specified query, comprising one or morewords.

At the completion of act 1205, example process 1200 proceeds to act1210, wherein the query is associated with one or more informationcategories. This may be accomplished in any of numerous ways. Forexample, a query may be associated with an information category based atleast in part on the query's content.

For example, in some embodiments, the textual output generated in act1205 may be processed to determine a semantic meaning for individualwords and/or combinations of words in the user-specified query, and/orof the overall meaning of the query, to attempt to identify the type ofinformation that the user seeks. For example, if a user-specified queryincludes the words “Starbucks® in Boston,” then act 1210 may includedetermining that the word “Starbucks®” represents a restaurant, and thatthe word “Boston” represents a location. This may be accomplished in anyof numerous ways. For example, any suitable natural language processingtechnique (of which numerous examples exist) may be employed todetermine the semantic meaning of individual words, and/or combinationsof words, in a user-specified query. The natural language processingmay, for example, include applying, to individual words and/orcombinations of words in a user-specified query, semantic tags thatindicate a meaning for one or more of the words. The semantic tags mayrelate to any of numerous types of information. For example, a semantictag may relate to software applications, books, movies, sports, tobusinesses, music, television, and/or other information types. Theembodiments of the invention that employ semantic tags are not limitedto employing these or any other particular types of tags. Of course,embodiments of the invention are not limited to employing any particularnatural language processing technique or applying semantic tags, as anysuitable technique, whether now known or later developed, may beemployed.

In some embodiments, the determination of one or more informationcategories to which a query relates and the behavior to be initiated inresponse is based (at least in part) on the semantic tags applied to thequery. For example, a collection of possible information categories andassociated behaviors may be defined, and a query may be associated withone or more of the information categories based on semantic tags thatare applied to the query. Using the example “Starbucks® in Boston” queryto illustrate, the collection of possible information categories mayinclude a “restaurant” information category and a “location” informationcategory, and queries to which the “restaurants” and “location” semantictags are applied may be associated with these information categories.One or more behaviors can be associated with this combination ofinformation categories, as discussed below. Of course, associating aquery with one or more information categories need not be performedusing a predefined collection of information categories and semantictags. For example, information categories and/or semantic tags may bedynamically defined, using any suitable technique(s). Embodiments of theinvention are not limited to any particular manner of implementation.

A user-specified query may be associated with multiple informationcategories that have different meanings. In this respect, the inventorshave recognized that while a user submitting a query may intend onemeaning for the content of the query, other meanings may also bepossible. For example, a user submitting the query “Fargo” may intend tolook for information relating to the city in North Dakota, but othersemantic meanings are also possible (e.g., the user may seek informationon the film). Thus, some embodiments of the invention enable auser-specified query to be associated with multiple informationcategories. For example, the word “Fargo” in the user-specified querymay cause the query be associated with one information category relatingto to geographic locations and another relating to movies. Thus, inaccordance with some embodiments of the invention, if it is determined,based on the content of a received query, that the content may have atleast a first semantic meaning or a second semantic meaning that isdifferent than the first semantic meaning, then a plurality of searchengines to which to submit a representation of the query may beidentified. The plurality of search engines may comprise a first searchengine identified based on the first semantic meaning of the content anda second search engine identified based on the second semantic meaningof the content. A user-specified query may be associated with anysuitable number of information categories, as embodiments of theinvention are not limited in this respect.

In some embodiments, a user-specified query may be associated with aninformation category using information other than the content of thequery. For example, the user's location, search history, browsinghistory, and/or other information may influence the association of aquery with one or more information categories, and/or the prioritizationof behaviors or actions taken if the query is associated with multiplecategories. Using the example query “Fargo” above to illustrate, if theuser's search and/or browsing history reveals that the user has a strongaffinity for films, then this information may cause the query “Fargo” tobe associated with the “movies” information category and not the“geographic locations” information category, or if associated with both,may treat the “movies” association as being more likely and therefore asa higher priority than the “geographic location” association indetermining one or more actions to take, as discussed below. Conversely,if location data indicates that the user is in North Dakota when thequery is submitted, then this information may cause the query “Fargo” tobe associated with the “geographic locations” information category andnot the “movies” information category, or if associated with both, maytreat the “geographic locations” association as a higher priority. Anyof numerous types of information may be considered when associating auser-specified query with an information category, as embodiments of theinvention are not limited in this respect.

At the completion of act 1210, example process 1200 proceeds to act1215, wherein a “behavior” for the identified one or more informationcategories is determined. In some embodiments, act 1215 may includeidentifying one or more search engines and/or other facilities makingavailable information relevant to the one or more information categoriesidentified in act 1210.

It should be appreciated that embodiments of the invention are notlimited to submitting user-specified queries to search engines providedby websites. A behavior defined for an information category may compriseinitiating and/or performing any of numerous types of actions, which mayor may not include identifying one or more search engines to which arepresentation of the user-specified query should be submitted. Forexample, a behavior defined for an information category may includeperforming one or more native actions. In this respect, the inventorshave recognized that some user-specified queries may reveal a desire onthe user's part to perform an action, in addition to or rather than tosearch for information. As discussed in greater detail below, initiatinga native action may include (as examples) launching an application, suchas a media player application, map application, etc., a service (e.g.,one which initiates a telephone call from a client device), and/or anyother natively executed facility. Some embodiments of the inventiondefine a behavior for a user-specified query which includes initiatingone or more native actions.

Identifying a behavior to be initiated in act 1215 may be performed inany of numerous ways. In some embodiments, a defined mapping associatesinformation categories with sets of behaviors, each set comprising oneor more native actions and/or one or more search engines to which acorresponding user-specified query should be submitted. The mapping mayalso define a prioritization among the behaviors. As discussed above,prioritization can be based on any suitable criteria (e.g., relevance,compensation provided by the search engine, etc.). A set of behaviorsmay include any suitable number of native actions and/or search engines(e.g., a set need not comprise a plurality of native actions or searchengines), as embodiments of the invention are not limited in thisrespect.

For example, a defined mapping may specify that user-specified queriesassociated with a “restaurant” and “location” information categoriescause a set of behaviors to be initiated which includes launching one ormore native actions and submitting the query to one or more searchengines previously determined to be appropriate recipients for queriesof the identified type. As a result, act 1215 may involve identifyingone or more native actions to initiate and/or one or more search enginesto which to submit the example user-specified query “Starbucks® inBoston” associated with the restaurant location information category inact 1210.

Table 1 below provides an illustrative mapping of various informationcategories to behaviors. Referring to the “Starbucks® in Boston” exampleto illustrate, Table 1 defines a mapping of user-specified queriesassociated with the “restaurant” and “location” information categoriesto a set of behaviors which includes launching a map application(“Maps,” shown in the “Tab 1 (Prime)” column of Table 1), submitting arepresentation of the query to a search engine offered by a websiteproviding restaurant reviews (“Yelp,” shown in the “Tab 2” column),submitting a representation of the query to a general-purpose searchengine so that telephone numbers included in search results may beidentified and presented (“Call Tab,” shown in the “Tab 3” column),submitting a representation of the query to a search engine offered by awebsite providing reference information (“Wikipedia,” shown in the “Tab4” column), submitting a representation of the query to a search engineprovided by a social networking website (“Twitter,” shown in the “Tab 5”column), and submitting a representation of the query to ageneral-purpose search engine that is identified by the user (“Gen.Search 1,” shown in the “Tab 6” column). Of course, Table 1 is anon-limiting example and is merely intended as an illustration of thetypes of behaviors that may be mapped to information categories.Embodiments of the invention are not limited to employing the mappingshown in Table 1, or indeed any mapping at all.

TABLE 1 Search Native Tab 1 Type Category 1 Category 2 Example App 1Engine Tab 7 Tab 5 Tab 3 (Prime) Tab 2 Tab 4 Tab 6 Application App buybuy angry birds Appl-Name-Buy N/A Gen. Search 1 YouTube Media Con.Wikipedia Images N/A Search Application App download download theAppl-Name- N/A Gen. Search 1 YouTube Media Con. Wikipedia Images N/ASearch free wi fi app for Download ipad Application App N/A free wi fiapp Appl-Name-NA N/A Gen. Search 1 YouTube Media Con. Wikipedia ImagesN/A Search Application iOS App N/A alarm clock for Appl-Product- N/AGen. Search 1 MacWorld.com Media Con. Wikipedia YouTube Images Searchipod NA (app reviews) Application iOS App N/A iphone app Appl-Product-N/A Gen. Search 1 MacWorld.com Media Con. Wikipedia YouTube ImagesSearch review Review (app reviews) Books Author Buy buy charlesBooks-Author- N/A Twitter Amazon Media Con. Local Retail Wikipedia Gen.Search 1 (download) dickens Buy Books Author N/A charles dickensBooks-Author- N/A Twitter Amazon Media Con. Local Retail Wikipedia Gen.Search 1 NA Books Author Photos/ charles dickens Books-Author- N/ATwitter Media Con. Images Wikipedia Amazon Gen. Search 1 Images imagesPhotos Books Author Play read charles Books-Author- N/A Twitter AmazonMedia Con. Local Retail Wikipedia Gen. Search 1 (locally) dickens PlayBooks Title Buy buy the girl with Books-Title-Buy N/A Twitter AmazonMedia Con. Local Retail Wikipedia Gen. Search 1 (download) the dragontattoo Books Title N/A the girl with the Books-Title-NA N/A TwitterAmazon Media Con. Local Retail Wikipedia Gen. Search 1 dragon tattooBooks Title Play read the girl with Books-Title-Play N/A Twitter AmazonMedia Con. Local Retail Wikipedia Gen. Search 1 (locally) the dragontattoo Books Title Reviews reviews of the Books-Title- N/A TwitterAmazon NYTimes.com Media Con. Local Retail Gen. Search 1 girl with theReview dragon tattoo Business Other Call call nuance direct callBusiness-Other- N/A Twitter Maps Call Yelp Wikipedia Gen. Search 1Search (if no dis- Call (launch app) ambiguation) Business OtherDirections directions to Maps App Business-Other- N/A Twitter Call TabYelp Maps Wikipedia Gen. Search 1 Search Target Directions BusinessOther Location target in boston Business-Other- N/A Twitter Call TabMaps Yelp Wikipedia Gen. Search 1 Search Location Business Other N/ATarget Business-Other- N/A Twitter Call Tab Yelp Maps Wikipedia Gen.Search 1 Search Info Business Other Near Me target near meBusiness-Other- N/A Twitter Call Tab Maps Yelp Wikipedia Gen. Search 1Search NearMe Business Other Photos/ photos of target Business-Other-N/A Twitter Maps Images Yelp Call Tab Gen. Search 1 Search Images PhotosBusiness Other Reviews review bestbuy Business-Other- N/A Twitter CallTab Yelp Maps Wikipedia Gen. Search 1 Search Review Business RestaurantsCall call starbucks direct call Business- N/A Wikipedia Twitter Call(launch Yelp Maps Gen. Search 1 Search (if no dis- Restaurant-Call app)ambiguation) Business Restaurants Directions directions to Maps AppBusiness- N/A Twitter Call Tab Yelp Maps Wikipedia Gen. Search 1 Searchstarbucks Restaurant- Directions Business Restaurants Locationsstarbucks in Business- N/A Twitter Call Tab Maps Yelp Wikipedia Gen.Search 1 Search boston Restaurant- Location Business Restaurants N/Astarbucks Business- N/A Twitter Call Tab Yelp Maps Wikipedia Gen. Search1 Search Restaurant-Info Business Restaurants Near Me starbucks nearBusiness- N/A Twitter Call Tab Maps Yelp Wikipedia Gen. Search 1 Searchme Restaurant- NearMe Business Restaurants Photos/ photos of Business-N/A Twitter Call Tab Images Yelp Maps Gen. Search 1 Search Imagesstarbucks Restaurant- Photos Business Restaurants Reservations starbucksBusiness- N/A Twitter Call Tab Yelp Maps Wikipedia Gen. Search 1 Searchreservations Restaurant- Reservation Business Restaurants Reviews reviewstarbucks Business- N/A Twitter Call Tab Yelp Maps Wikipedia Gen. Search1 Search Restaurant- Review Events N/A N/A local events EventSearch- N/ATwitter Zvents.com Eventful LiveNation Stubhub Gen. Search 1 NA-NAGeneral N/A N/A how many Default N/A YouTube Twitter Gen. Search 1Wikipedia Media Con. Images Web Search organs are in the human body?Maps N/A Directions directions from Maps App MapSearch-NA- N/A N/AImages Maps Gen. Search 1 Wikipedia N/A boston to Directions burlingtonMaps N/A Locations map of florida MapSearch-NA- N/A N/A Images Maps Gen.Search 1 Wikipedia N/A Location Maps N/A N/A maps MapSearch-NA- N/A N/AImages Maps Gen. Search 1 Wikipedia N/A NA Movies Actor/ Buy robert deniro Movies- N/A Amazon IMDB fandango Media Con. Wikipedia Gen. Search 1Director (theater) tickets ActorDirector- Tickets Movies Actor/ N/A TomHanks Movies- N/A Images Wikipedia IMDB Media Con. Twitter Gen. Search 1Director ActorDirector- Info Movies Actor/ Photos/ tom hanks Movies- N/Afandango IMDB Images Wikipedia Media Con. Gen. Search 1 Director Imagesimages ActorDirector- Photos Movies Actor/ Play play brad pitt Movies-N/A Twitter IMDB Media Con. Wikipedia Images Gen. Search 1 Director(locally) movies ActorDirector- Play Movies N/A Info movieMovies-NA-Info N/A Media Con. Twitter IMDB fandango rottentomatoes Gen.Search 1 information Movies N/A Review movie reviews Movies-NA- N/Afandango Twitter rottentomatoes IMDB Media Con. Gen. Search 1 ReviewMovies N/A Schedule movie schedules Movies-NA- N/A Media Con. Twitterfandango IMDB rottentomatoes Gen. Search 1 Schedule Movies Title PlayPlay avatar Play media Movies-Title- N/A rottentomatoes Twitter MediaCon. IMDB Wikipedia Gen. Search 1 (locally) Play Movies Title Playstream avatar Movies-Title- N/A rottentomatoes Twitter Media Con. IMDBWikipedia Gen. Search 1 (stream) Stream Movies Title Schedule showtimesfor Movies-Title- N/A rottentomatoes Twitter fandango IMDB WikipediaGen. Search 1 avatar Schedule Movies Title Song seabiscuit themeMovies-Title- N/A rottentomatoes Twitter Media Con. IMDB Wikipedia Gen.Search 1 song Song Movies Title [in Buy tickets for Movies-Title- N/ATwitter IMDB fandango rottentomatoes Wikipedia Gen. Search 1 theaters](theater) inception Tickets Movies Title [in N/A inception Movies-Title-N/A Twitter fandango IMDB rottentomatoes Wikipedia Gen. Search 1theaters] Info Movies Title [in Photos/ inception photos Movies-Title-N/A fandango IMDB Images rottentomatoes Wikipedia Gen. Search 1theaters] Images Photos Movies Title [in Review inception reviewMovies-Title- N/A Media Con. IMDB rottentomatoes fandango Wikipedia Gen.Search 1 theaters] Review Movies Title [not in Buy buy avatarMovies-Title- N/A IMDB Amazon Media Con. Local Retail rottentomatoesGen. Search 1 theaters] (download) Buy Movies Title [not in N/A avatarMovies-Title- N/A Twitter Media Con. IMDB rottentomatoes Wikipedia Gen.Search 1 theaters] Info Movies Title [not in Photos/ avatar photosMovies-Title- N/A Gen. Search 1 IMDB Images rottentomatoes WikipediaGen. Search 1 theaters] Images Photos Movies Title [not in Review avatarreview Movies-Title- N/A Twitter IMDB rottentomatoes Wikipedia MediaCon. Gen. Search 1 theaters] Review Music Artist Buy buy lady gagaMusic-Artist- N/A LiveNation Amazon Media Con. Local Retail WikipediaGen. Search 1 (download) music Buy Music Artist N/A lady gagaMusic-Artist- N/A LiveNation Twitter Media Con. Wikipedia Images Gen.Search 1 Info Music Artist N/A joe lean on Music-Artist- N/A Gen. Search1 IMDB Media Con. Wikipedia Twitter Images goodfellas Movie Music ArtistPhotos/ show me a photo Music-Artist- N/A LiveNation Twitter ImagesMedia Con. Wikipedia Gen. Search 1 Images of lady gaga Photos MusicArtist Play play lady gaga Play media Music-Artist- N/A LiveNationTwitter Media Con. Wikipedia Images Gen. Search 1 (locally) in back-Play ground Music Artist Radio stream lady gaga Pandora Music-Artist-N/A LiveNation Twitter Media Con. Wikipedia Images Gen. Search 1(stream) songs Stream Music Artist Tickets lady gaga ticketsMusic-Artist- N/A Media Con. Twitter LiveNation StubHub Wikipedia Gen.Search 1 (buy) Tickets Music Song/Album Buy buy album love Music- N/ATwitter Amazon Media Con. Local Retail Wikipedia Gen. Search 1(download) Album/Song- Buy Music Song/Album N/A macarena Music-Song-InfoN/A YouTube Twitter Media Con. Wikipedia Gen. Search 1 N/A MusicSong/Album Play play macarena Play media Music- N/A N/A Twitter MediaCon. Wikipedia Gen. Search 1 N/A (locally) in back- Album/Song- groundPlay Music Song/Album Radio stream album Pandora Music- N/A N/A TwitterMedia Con. Wikipedia Gen. Search 1 N/A (stream) love love loveAlbum/Song- Stream Music Artist N/A lady gaga videos Videos-Artist- N/ALiveNation YouTube Media Con. Wikipedia Twitter Gen. Search 1 VideosInfo Music Artist Play play lady gaga Play media Videos-Artist- N/ALiveNation YouTube Media Con. Wikipedia Twitter Gen. Search 1 Videos(locally) music videos in back- Play ground Music Artist Play streamlady gaga VEVO Videos-Artist- N/A LiveNation YouTube Media Con.Wikipedia Twitter Gen. Search 1 Videos (stream) music videos StreamMusic Song Play play the music Play media Videos-Song- N/A TwitterYouTube Media Con. Wikipedia Gen. Search 1 N/A Videos (locally) videofor in back- Play macarena ground Music Song Play stream the music VEVOVideos-Song- N/A Twitter YouTube Media Con. Wikipedia Gen. Search 1 N/AVideos (stream) video for Stream macarena News N/A N/A Haiti earthquakeNews-NA-NA N/A YouTube Twitter CNN NYTimes.com Images Gen. Search 1 NewsUser N/A USAToday News-SpecSite- Gen. Images NYTimes.com User CNNTwitter YouTube Specified NA Search 1 Specified Site Site Product LocalRetail N/A call of duty Product-Game- N/A Twitter Nextag Amazon LocalRetail IGN Gen. Search 1 Search Info Product Local Retail N/A iphoneProduct-NA-Info N/A Twitter Nextag Amazon Local Retail Images Gen.Search 1 Search Product Local Retail N/A iphone review Product-NA- N/ATwitter Nextag Amazon Local Retail Images Gen. Search 1 Search ReviewProduct Local Retail Reviews call of duty Product-Game- N/A TwitterAmazon IGN Local Retail Nextag Gen. Search 1 Search review ReviewProduct N/A N/A ipod near by Product- N/A Gen. Search 1 Amazon LocalRetail Nextag Twitter Images Search LocalRetail-NA Product N/A Photos/iphone photos Product-NA- N/A Nextag Amazon Images Local Retail TwitterGen. Search 1 Search Images Photos Social Facebook N/A marc regan onURL-NA-NA N/A Gen. Search 1 MySpace Facebook Twitter Images N/A Searchfacebook Social MySpace N/A marc regan on URL-NA-NA N/A Gen. Search 1Twitter MySpace Facebook Images N/A Search myspace Social Twitter N/Amarc regan on URL-NA-NA N/A Gen. Search 1 MySpace Twitter FacebookImages N/A Search twitter Sports Athlete Biography michael jordanSports-Athlete- N/A Gen. Search 1 Twitter Wikipedia ESPN Images N/Abiography Bio Sports Athlete N/A michael jordan Sports-Athlete- N/A Gen.Search 1 Twitter ESPN Wikipedia Images N/A NA Sports Athlete Photos/photos of Sports-Athlete- N/A Gen. Search 1 Twitter Images ESPNWikipedia N/A Images michael jordan Photos Sports Athlete Statisticsmichael jordan Sports-Athlete- N/A Gen. Search 1 Twitter ESPN [playerWikipedia Images N/A statistics Stats stats] Sports Sport Type N/A NBASports- N/A Twitter Wikipedia ESPN Gen. Search 1 Images N/ASportType-Info Sports Sport Type News NBA news Sports- N/A TwitterWikipedia ESPN [league Gen. Search 1 Images N/A SportType-Info news]Sports Sport Type Photos/ NHL photos Sports- N/A Twitter ESPN ImagesWikipedia Gen. Search 1 N/A Images SportType- Photos Sports Sport TypeSchedule NBA schedule Sports- N/A Twitter Wikipedia ESPN [league Gen.Search 1 Images N/A SportType-Info schedule] Sports Sport Type ScoresNBA scores Sports- N/A Twitter Wikipedia ESPN [league Gen. Search 1Images N/A SportType-Info scores] Sports Sport Type Standings NBAstandings Sports- N/A Twitter Wikipedia ESPN [league Gen. Search 1Images N/A SportType-Info standings] Sports Sport Type Statistics NBAstats Sports- N/A Twitter Wikipedia ESPN [league Gen. Search 1 ImagesN/A SportType-Info stats] Sports Sport Type Tickets football ticketsSports- N/A Twitter ESPN StubHub TicketMaster Gen. Search 1 N/ASportType- Tickets Sports Team N/A dallas cowboys Sports-Team- N/A Gen.Search 1 Wikipedia ESPN Twitter Images N/A Info Sports Team News dallascowboys Sports-Team- N/A Gen. Search 1 Wikipedia ESPN [team TwitterImages N/A news Info news] Sports Team Photos/ boston bruinsSport-SportType- N/A Twitter ESPN Images Wikipedia Gen. Search 1 N/AImages photos Photos Sports Team Schedule dallas cowboys Sports-Team-N/A Gen. Search 1 Wikipedia ESPN [team Twitter Images N/A schedule Infoschedule] Sports Team Scores dallas cowboys Sports-Team- N/A Gen. Search1 Wikipedia ESPN [team Twitter Images N/A scores Info scores] SportsTeam Standings dallas cowboys Sports-Team- N/A Gen. Search 1 WikipediaESPN [team Twitter Images N/A standings Info standings] Sports TeamStatistics dallas cowboys Sports-Team- N/A Gen. Search 1 Wikipedia ESPN[team Twitter Images N/A stats Info stats] Sports Team Tickets Ticketsfor dallas Sports-Team- N/A Twitter ESPN [team] StubHub TicketMasterGen. Search 1 N/A cowboys games Tickets Sports Team Tickets find mechicago Sports-Team- N/A Twitter ESPN [team] StubHub TicketMaster Gen.Search 1 N/A blackhawks Tickets tickets Television Actor/ Buy buy DavidTv-Actor-Buy N/A IMDB Amazon Media Con. Local Retail Gen. Search 1 N/ADirector (downoad) Letterman shows Television Actor/ Buy (store) buyDavid Tv-Actor-Buy N/A IMDB Media Con. Amazon Local Retail Gen. Search 1N/A Director Letterman on DVD Television Actor/ N/A David LettermanTv-Actor-Info N/A Gen. Search 1 IMDB Wikipedia Twitter Media Con. ImagesDirector Television Actor/ Photos/ David Letterman Tv-Actor-Photos N/AGen. Search 1 Twitter Images Wikipedia IMDB Media Con. Director Imagesphotos Television Actor/ Play play David Tv-Actor-Play N/A Twitter IMDBMedia Con. Wikipedia Gen. Search 1 N/A Director (locally) Lettermanshows Television Actor/ Play stream David Tv-Actor-Stream N/A TwitterIMDB Media Con. Wikipedia Gen. Search 1 N/A Director (stream) Lettermanshows Television Title Buy buy married with Tv-Title-Buy N/A IMDB AmazonMedia Con. Local Retail Twitter Gen. Search 1 (downoad) childrenTelevision Title Buy (store) buy married with Tv-Title-Buy N/A IMDBMedia Con. Amazon Local Retail Twitter Gen. Search 1 children on blu-ray Television Title N/A married with Tv-Title-Info N/A IMDB AmazonMedia Con. Local Retail Twitter Gen. Search 1 children Television TitlePhotos/ married with Tv-Title-Photos N/A IMDB Media Con. Images TwitterGen. Search 1 N/A Images children photos Television Title Play playmarried Tv-Title-Play N/A IMDB Amazon Media Con. Local Retail TwitterGen. Search 1 (locally) with children Television Title Play streammarried Tv-Title-Stream N/A IMDB Amazon Media Con. Local Retail TwitterGen. Search 1 (stream) with children Television Title Song punkybrewster Tv-Title-Song N/A N/A IMDB Media Con. Amazon Gen. Search 1 N/Atheme song User Content call of duty 4 URL-NA-NA Tab 6 Tab 4 Tab 2 UserTab 1 Tab 3 Tab 5 Specified Type review on Specified Site (prime) SiteKNOWN joystiq.com User Content amazon Phillips razor on URL-NA-NA N/AGen. Search 1 eBay Amazon Local Retail Images N/A Specified Type AmazonSite UNKNOWN User Content buy Phillips razor on URL-NA-NA N/A Gen.Search 1 Amazon Buy Local Retail eBay Images Specified Type Buy.com SiteUNKNOWN User Content ebay Phillips razor on URL-NA-NA N/A Gen. Search 1Amazon eBay Local Retail Images N/A Specified Type Ebay Site UNKNOWNUser Content Google directions to Google maps URL-NA-NA Tab 7 Tab 5 Tab3 Google maps Tab 2 Tab 4 Tab 6 Specified Type maps - walmart on SiteUNKNOWN directions Google maps User Content Google Boston on URL-NA-NATab 7 Tab 5 Tab 3 Google maps Tab 2 Tab 4 Tab 6 Specified Type maps -Google maps Site UNKNOWN other User Content lastfm lady gaga on lastfmURL-NA-NA Tab 6/7 Tab 4 Tab 2 User Tab Tab 3 Tab 5 Specified TypeSpecified Site 1(prime) Site UNKNOWN User Content pandora lady gaga onPandora URL-NA-NA Tab 6/7 Tab 4 Tab 2 User Tab Tab 3 Tab 5 SpecifiedType pandora Specified Site 1(prime) Site UNKNOWN User Content hotelreviews on URL-NA-NA Tab 6/7 Tab 4 Tab 2 User Tab Tab 3 Tab 5 SpecifiedType livejournal.com Specified Site 1(prime) Site UNKNOWN Videos TitleBuy buy video my Videos-Title-Buy N/A YouTube eBay Media Con. AmazonLocal Retail Gen. Search 1 fair lady Videos Title Stream stream video myVideos-Title- N/A eBay YouTube Media Con. Amazon Local Retail Gen.Search 1 fair lady Stream Videos N/A N/A dog tricks videos Videos-NA-NAN/A Amazon Twitter YouTube Media Con. Gen. Search 1 N/A (generic)Weather N/A N/A Current weather Weather-NA-NA N/A N/A wunderground.comaccuweather.com weather Gen. Search 1 N/A

As can be seen from Table 1, user-specified queries associated withdifferent information categories may be mapped to different sets ofbehaviors. For example, a user-specified query associated with the“Title [in theaters]” and “Buy [Theater]” information categories may besubmitted to a set which includes a search engine provided by a websiteenabling users to purchase movie tickets (“fandango,” shown in the “Tab1 (Prime)” column of Table 1), a search engine offered by a websiteproviding movie reviews (“rottentomatoes,” shown in the “Tab 2” column),a search engine offered by a website providing movie information(“IMDB,” shown in the “Tab 3” column), a search engine offered by awebsite providing reference information (“Wikipedia,” shown to in the“Tab 4” column), a search engine provided by a social networking website(“Twitter,” shown in the “Tab 5” column), and a general-purpose searchengine identified by the user (“Gen. Search 1,” shown in the “Tab 6”column). Any suitable number and type of behaviors may be mapped to aset of information categories, as embodiments of the invention are notlimited in this respect.

It should be appreciated that embodiments of the invention are notlimited to employing a pre-defined mapping of information categories tosets of search engines and/or native actions. Any suitable technique(s)may be employed to determine a set of behaviors appropriate for auser-specified query associated with a particular information category.For example, if a set of behaviors includes submitting a query to one ormore search engines, the search engine(s) may be identified dynamically,such as based on the content of the query, and/or other information. Forexample, as described further below, the presence and/or accuracy oflocation data provided with a user-specified query may be used at leastpartially to determine which search engine(s) are identified in act1215.

It should be appreciated that identifying search engines to which auser-specified query should be submitted, in accordance with someembodiments of the invention, offers the opportunity to make theparticular type of information determined to be that which the userseeks available to him/her more quickly than conventional approaches mayallow. Using the above example query “Starbucks® in Boston” toillustrate, a conventional approach to searching for Starbucks®'locations in Boston may entail the user submitting the same examplequery to a general-purpose search engine. While the results returned bythe general-purpose search engine may include pages provided by similarsites to those identified in the example above (i.e., by a siteproviding access to map data, a site providing restaurant reviews, asite providing reference information, etc.), there is no guarantee thatresults from these sites will be included in the results. Further, evenif the results include pages from sites similar to those identified inthe example, the user may be forced to sift through other results thatare unrelated to the particular information that he/she seeks. Forexample, a general-purpose search engine may respond to the query“Starbucks® in Boston” by generating results including links to sitesproviding information on other restaurants in Boston, information on theStarbucks® corporation, information on the city of Boston, etc. None ofthis is the to information the user seeks, but the user must review (atleast superficially) all of it to locate the type of information he/sheultimately seeks. In addition, if the user is interested in informationfrom a website providing map data, the user must select that link in thesaved search results restored, as these results are not directlypresented. Thus, conventional approaches may require a user to expendunnecessary amounts of time and mental energy to locate information of aparticular type. By contrast, embodiments of the invention may identify,upon receiving a user's query, the particular type of information thatthe user seeks, and attempt to immediately present information of thattype to the user (e.g., based on an estimated similarity between thecontent of the query, and/or other information, and content madeavailable by one or more search engines). As a result, embodiments ofthe invention may reduce the time and cognitive load associated withlocating a particular type of information. Using the example above toillustrate, the user may be immediately presented with results from aplurality of search engines, for instance, map data showing thelocations of Starbucks® restaurants in Boston, reviews of variousStarbucks® locations in Boston, etc.

It should be appreciated that by identifying a plurality of searchengines to which queries for information of a particular type should besubmitted, embodiments of the invention can be aggressive in attemptingto immediately present a specific type of information the user isbelieved to seek, because other less specific information cansimultaneously be provided as a fallback option in case the aggressive,specific approach turns out to misapprehend the user's intent. As anexample, for queries associated with a particular information category,embodiments of the invention may identify one or more “most aggressive”options (e.g., provided one or more sites making available specializedinformation that is closely related to a particular informationcategory). In the example given above, these options may be the siteproviding access to map data and/or restaurant reviews. As described infurther detail below, results generated by the most aggressive option(s)may be presented more prominently than results generated by other, lessspecific options. In some embodiments, the freedom to presentspecialized information prominently in response to a search is affordedby the other, less aggressive search engines in the identified set,which provide fallback options in case the specialized information isnot the type that the user seeks. In the example above, these optionsare provided by the search engines provided by sites making availablereference information, to etc. Thus, if the primary options do notprovide the type of information the user seeks, the alternatives alsopresented may provide more generalized information that may encompassthe type of information which the user seeks. In addition, even if themost aggressive option provides the type of information that the userinitially sought, these fallback options may provide access to otherinformation related to the specific type of information which the mostaggressive option provides, but which the user may not have thought tosearch for, or may have intended to also search for after completing aninitial search.

In some embodiments, identifying a set of search engines to which auser-specified query should be submitted may involve identifying one ormore search engines for each of multiple potential meanings of thecontent of the query. In this respect, as noted above, a user-specifiedquery may be associated in act 1210 with more than one informationcategory, perhaps because the content of the user's query is identifiedas having more than one potential meaning. If a user-specified query isassociated with more than one information category in act 1210, thenidentifying in act 1215 a behavior (e.g., a set of search engines towhich the query should be submitted) may entail any of numerous actions.For example, in some embodiments, act 1215 may include identifying whichpotential meaning is most likely to be the meaning intended by the user.This may be accomplished in any of numerous ways. For example, theuser's location, search history, browsing history, and/or otherinformation may be analyzed to determine a probability that the userintended each potential meaning, using any of numerous techniques. Usingthe “Fargo” example query given above to illustrate, if the user hasrecently searched for information on other films and is not currentlylocated in North Dakota, then it may be determined that it is moreprobable that the user intended to search for information on the filmFargo, and less probable that the user intended to search forinformation on the city. As a result, the set of search enginesidentified as being the ones to which the query should be submitted,and/or the ones from which results should be most prominently displayed,may be the ones associated with the movies information category ratherthan the geographic locations information category.

As mentioned above, embodiments of the invention are not limited todetermining which potential meaning is most likely and identifying onlya set of search engines to which the query should be submitted on thisbasis, as queries can be submitted to different search engines based onalternative semantic meanings. Thus, in some embodiments, for theexample query “Fargo,” a set of search engines may, for example, beidentified which includes a first search engine provided by a websitethat makes available information on films, a second search engineprovided by a website that makes available information on geographiclocations, a third, general-purpose search engine, etc., so that morethan one potential meaning may be represented in the results generated.In alternate embodiments, if multiple potential meanings for auser-specified query are identified, only general-purpose search enginesmay be identified as that to which the user's query should be submitted,since more than one potential meaning may be represented in the resultsgenerated by the general-purpose search engine.

Further, if a user-specified query cannot be associated with anyinformation categories in act 1210 (e.g., because no semantic tags areapplied to the query, so that the query is not associated with aninformation category), then act 1215 may include identifying one or moregeneral-purpose search engines as the search engine(s) to which theuser's query should be submitted. Embodiments of the invention mayprocess a query which cannot be associated with an information categoryin any suitable fashion.

In some embodiments, the number of search engines identified in act 1215may differ depending on the information category or categories to whichthe query is mapped, and the number and type of search engines relevantthereto. For example, act 1215 may include, for a user-specified querymapped to the restaurant locations information category in act 1210,identifying a first quantity of search engines to which the query shouldbe submitted. For a user-specified query mapped to the geographiclocations information category in act 1210, act 1215 may includeidentifying a different quantity of search engines to which the queryshould be submitted. Queries mapped to multiple information categories,or mapped to no information categories, may have different numbers ofsearch engines identified. Any suitable number of search engines may beidentified for any user-specified query, as embodiments of the inventionare not limited in this respect.

In some embodiments, a user may specify one or more of the searchengines to which one or more queries may be submitted. For example, theuser may specify that certain queries (e.g., all queries, those whichare mapped to particular information categories, etc.) be submitted toone or more specified search engines. The user may specify such searchengine(s) in any suitable manner, e.g., via direct input (e.g., to aclient device via a keyboard, touch screen, and/or other inputfacility), and this information may be stored by the application programon client device 250 and/or external computer 275. In some embodiments,input specifying one or more search engines may be supplied when theuser provides a search query to the application program. In otherembodiments, the input may be supplied once by the user and stored, andeach time a user-specified query is received the information may beaccessed to determine whether there are previously-specified searchengines to which the query is to be submitted. Embodiments of theinvention may store and/or retrieve one or more specified search enginesin any of numerous ways.

A user may also specify within the query one or more search engines towhich the query should be submitted. For example, the user may submitthe query “give me the Wikipedia page for George Washington,” or “giveme the Yelp review for John Doe's Bar and Grill,” and embodiments of theinvention may provide for the query being submitted to the identifiedsearch engine. This may be accomplished in any of numerous ways. In someembodiments, queries that include the name of a particular search enginemay be mapped to a particular information category in act 1210 (e.g.,based on the presence of the word identifying the search engine in thequery), and a behavior may be identified for that information categorythat includes submitting the query to the identified search engine. Inother embodiments, the name of a search engine within a user-specifiedquery may be identified without mapping the query to an informationcategory. Embodiments of the invention may identify a search engine towhich a query is submitted using any suitable technique(s).

As noted above, in some embodiments, act 1215 need not include onlydetermining a set of search engines to which a user-specified queryshould be submitted. For example, in some embodiments, a behavior thatis identified in act 1215 for a user-specified query may include theinitiation of one or more native actions on client device 250, either inaddition to or instead of determining one or more search engines for aquery. In this respect, the inventors have recognized that someuser-specified queries may reveal a desire to perform an action inaddition to or rather than search for to information. For example, auser-specified query that includes the words “play I Got a Feelingvideo” may indicate a desire to play a video, rather than to search forinformation on the song “I Got a Feeling.” As a result, some embodimentsof the invention may define a behavior for a query which includesinitiating that action. For example, the action may include launching amedia player application on the user's device to play a song or video(as in the example above), and/or performing one or more other actions.Thus, in some embodiments, an external computer (e.g., external computer275, shown in FIG. 2) may, upon receiving a query comprising contentfrom a device, determine based at least in part on the content of thequery that an application is to be launched on the device. The externalcomputer may cause the device to launch the application using at leastsome information determined from the content of the query. In someembodiments, a client device (e.g., client device 250, shown in FIG. 2)may receive a free-form query from a user, transfer a representation ofthe query to at least one computer, and receive from the at least onecomputer at least one instruction to launch an application on thedevice.

In some embodiments, a behavior that is defined for a user-specifiedquery may include both initiating a native action and also initiating asearch for information to one or more search engines. For example, if auser-specified query appears to indicate a desire to initiate an action,the action may be initiated, and a query may be submitted to one or moreidentified search engines. This may, for example, be preferable incertain instances to merely initiating the native action, since the usermay have intended to search for information rather than initiate anaction, so that merely initiating the action without also initiating asearch may confuse and/or frustrate the user. Example ways in which auser may switch between a native action (e.g., a natively executingapplication) and search results are described in detail below.

In some embodiments, a behavior defined for a user-specified query mayinclude instructing a client device to perform one or more of a set ofalternative actions. In this respect, external computer 275, whichperforms the act 1215 in the example process 200, may not know whatapplications are installed on client device 250, and thus may not knowwhat applications are available to launch. In some embodiments, anordered list of actions (e.g., applications) may be specified by theexternal computer 275 and received by the client device. When theordered list is received, the client device may first to attempt toperform the first action (e.g., launch a first application) on the list,and if the action cannot be performed (e.g., because the firstapplication is not installed on the device), attempt to perform thesecond action (e.g., launch the second application) on the list, untileither the client device is able to perform a specified action or thelist of alternatives is exhausted. Of course, embodiments of theinvention that relate to initiating a native action are not limited tosuch an implementation. For example, an ordered list need not bespecified. Rather, in some embodiments, a native action may be specifiedto the client device in generic terms (e.g., “launch a media player”),and the client device may choose from the applications installed whichone to launch. In other embodiments, the external computer may beinformed of the applications installed on the device (e.g., viainformation provided with a user-specified query or in any othersuitable way), so that specific instructions may be given to theapplication on the client device about which application(s) to launch.Embodiments of the invention that relate to initiating a native actionare not limited to any particular implementation.

At the completion of act 1215, process 1200 completes.

Referring again to FIG. 2, at the completion of act 207 wherein the typeof information and/or action the user seeks is identified (e.g., usingany of the above-described techniques), example process 200 proceeds toact 209, wherein information is generated by external computer 275. Thismay be performed in any of numerous ways, and the information may takeany of various forms (e.g., depending on the behavior identified in act1215). For example, act 209 may include generating a representation ofthe user-specified query received in act 1205 for submission to one ormore search engines identified in act 1215, generating instructions toinitiate an action on the client device identified in act 1215, and/orgenerating any of numerous other forms of information, as embodiments ofthe invention are not limited in this respect.

If a behavior identified in act 1215 includes submitting arepresentation of the user-specified query to one or more searchengines, then act 209 may include external computer 275 generatinginstructions specifying the search engine(s) and a query to be submittedto each. In some embodiments, a query to be submitted to a search enginemay be specified in the form of a Universal Resource Locator (URL)having encoded therein a representation of the query, appropriatelyformatted for that search engine. For example, one or more words in auser-specified query may be encoded in a URL formatted for a searchengine. If a word in a user-specified query is susceptible to multiplerepresentations (e.g., a state name, which is commonly abbreviated),then act 209 may include modifying the word to conform to the searchengine's requirements. For example, act 209 may involve modifying theword “Massachusetts” in a user-specified query to be encoded as “MA”within a URL. Any suitable types of modification or other processing maybe performed, as embodiments of the invention are not limited in thisrespect. The modification or other processing may, in some embodiments,be performed by external computer 275. However, embodiments of theinvention are not limited to such an implementation, as the modificationor other processing may be performed by the client device 250, or byboth client device 250 and external device 275.

FIG. 9 depicts example URLs that may be generated for three differentsearch engines for the user-specified query “Miles Davis.” Of course,embodiments of the invention are not limited to employing the formatdepicted for any search engine, whether or not the search engine isrepresented in FIG. 9.

Some search engines cannot be queried with a simple URL. For example,for some search engines, the client first establishes a connection orsession with the search engine, and then sends the query to the searchengine. If such a search engine is identified as a search engine to bequeried, a query based on the content of the user-supplied query may begenerated in a format suitable for that search engine. Like the URLs,the query for such a search engine may be generated by the applicationprogram on the client device and/or by a computer external to the clientdevice (e.g., server 605).

If a behavior identified in act 1215 includes initiating a nativeaction, then act 209 may include external computer 275 generatinginstructions specifying the actions to be initiated. As noted above, theinstructions may, for example, specify a particular application to belaunched (e.g., using a Uniform Resource Identifier, or URI, identifyingthe application and parameters identifying content which the applicationshould process). Alternatively, the instructions may specify a pluralityof applications (e.g., in an ordered list), from which the client devicemay select one to launch, in any suitable manner (e.g., to by attemptingto launch each application specified until an application installed onthe device is found).

In another embodiment, the instructions may specify a desired behavior(e.g., an action to be initiated) without specifying an application toachieve it, thereby leaving it to the client to identify an installedapplication corresponding to the specified behavior. It should beappreciated that embodiments of the invention are not limited togenerating instructions specifying that only a single behavior actionshould be initiated, as any suitable number of actions may be initiated.Embodiments of the invention are not limited in this respect.

Act 209 may include generating instructions specifying that both anative action is to be initiated and that a representation of theuser-specified query is to be submitted to one or more search engines.These instructions may specify that these tasks be performed in any ofnumerous ways. For example, act 1220 may include generating instructionsspecifying that an application is to be launched, and that the searchresults should be displayed as a backup option if the correspondingapplication is stopped by the user after launch. Alternatively, theinstructions may specify that initiation of the native action should bepresented as an option which the user may select (e.g., together with anoption of having search queries submitted, among search results returnedby identified search engines, etc.). Any of numerous implementations arepossible. As another alternative, if the client device allows multipleapplications to be open simultaneously, the native application can belaunched (e.g., in one window) while the search results are returnedsimultaneously (e.g., in another window).

In some embodiments, act 209 may include generating instructionsinstructing the client device to provide additional information to theexternal computer 275. In one example, the instructions generated in act209 may include a request to provide more accurate location information,since the device's location may drive whether certain behaviors (e.g.,submission of queries to certain search engines, and/or includingcertain search parameters in a query) are appropriate. In this respect,the location information provided with a user-specified query in act 203may be at any degree of specificity. For example, the location data mayonly indicate the location of a cell tower with which the devicecommunicates (and be accurate to within about 2,000 meters), mayidentify a to rough location as supplied by Global Positioning Services(GPS) satellites (and be accurate to within about 100 meters), oridentify a more exact location identified by GPS (and be accurate towithin about 10 meters). Different searches may require differentdegrees of precision in location data to be performed. For example, if auser-specified query includes the words “subway near me,” and theaccurate location provided with the query is only accurate to withinabout 2,000 meters, then more specific or accurate location data may bedesired before a response to the query may be given with a desireddegree of accuracy.

An example process 1300 via which an external computer 275 may determinewhether location data supplied by a client device is sufficientlyspecific, and if not may request more specific location data, is shownin FIG. 13. Example process 1300 may, for example, be performed after auser-specified query is received by external computer 275 (e.g., afteract 205 in example process 200, and prior to or after act 207).

At the start of example process 1300, external computer 275 determineswhether the location data received is specific enough in act 1303. Thisdetermination may be made in any of numerous ways, based on any ofnumerous criteria. For example, as noted above, external computer 275may make this determination based upon analyzing the content of theuser's query (e.g., to determine whether the user is searching forlocation-specific information), based on predefined specificityrequirements of one or more particular search engines identified asthose to which a representation of the query is to be submitted,immediately upon receiving the query (e.g., the external computer mayrequest location data for all queries be provided at at least at athreshold level of granularity), or in any other suitable fashion.

If it is determined that the location data is sufficiently specific,then example process 1300 terminates by proceeding to act 207 (FIG. 2).If it is determined that the location data is not sufficiently specific,then example process 1300 proceeds to act 1305, wherein a request formore specific location information is generated, and sent to the clientdevice in act 1307.

The client device receives the request in act 1309, and in act 1311determines whether more specific location data than that which wasinitially provided is available. More specific location information mayhave become available for any of numerous reasons. For example, when theuser-specified query was initially submitted to the external computer inact 203, the client device may have been in the process of acquiringmore specific location data (e.g., from its GPS system), and thatinformation may now be available. If more specific location data isavailable, then example process 1300 proceeds to act 1313, wherein theinformation is sent to external computer 275. The information isreceived by the external computer in act 1315, and example process 1300terminates by proceeding to act 207. In act 207, the process of FIG. 2may employ the more specific location data in any suitable way, oneexample of which includes identifying one or more search engines towhich a user-specified query should be submitted.

If it is determined in act 1311 that more specific location data has notbecome available, then example process 1300 proceeds to act 1317,wherein the client device attempts to obtain more specific information.For example, act 1317 may include client device 250 attempting toactivate its GPS system, if one is available and not already activated.Client device 250 then begins a predetermined timeout period in act1319. In this respect, the timeout period may be established to ensurethat only a predetermined period elapses before the client device eitherobtains more specific location data and sends it to external computer275, or informs external computer 275 that more specific location datais not available.

Example process 1300 then proceeds to act 1321, wherein it is determinedwhether the timeout period has elapsed. If not, a determination is madein act 1325 of whether more specific location data has been received inresponse to the request issued in act 1317. If not, the process returnsto act 1321, and then continues as described above until either thetimeout period expires or more specific location data is received. Ifthe timeout period elapses, then the client device sends an indicationthat more specific location data is not available in act 1323. If morespecific location data is obtained, then the client device sends themore specific data to the external computer in act 1327. The externalcomputer receives in act 1315 either the indication that more specificdata is not available or more specific data, and then process 1300proceeds to act 207, as described above with reference to FIG. 2. Forexample, act 207 may employ the more specific location data sent in act1327, or the indication that more specific location data is notavailable sent in act 1323, in identifying one or more search engines towhich a user-specified query should be submitted.

It should be appreciated that example process 1300 is but one example ofa technique which may be employed to obtain specific location data, andthat such data may be obtained using any suitable technique.

The availability or absence of specific location data may drive any ofnumerous types of system behavior. For example, for certain types ofqueries, the absence of sufficiently specific location data may alterthe search engines to which a user-specified query is submitted. Usingthe example “subway near me” query to illustrate, if the only locationdata that is available is accurate to within 2,000 meters, then thequery may not be submitted to a search engine that provides access tomap data, since any map generated may not provide information useful tothe user. Rather, the query may instead, for example, be submitted to asearch engine provided by a public transit authority website and one ormore general-purpose search engines, so that the user may locate anearby subway station using information provided by those informationsources rather than having one located automatically based on locationdata.

Referring again to FIG. 2, after the external computer has generatedinformation in act 209, example process 200 proceeds to act 211, whereinexternal computer 275 transmits the generated information to clientdevice 250. This may be performed in any suitable way, using anysuitable communications infrastructure and/or technique(s). Exampleprocess 200 then proceeds to act 213, wherein client device 250 receivesthe instructions.

It should be appreciated that although example process 200 illustratesexternal computer 275 determining the type(s) of information the userseeks and/or action(s) or behaviors the user wishes to perform,embodiments of the invention are not limited to being implemented inthis manner. For example, client device 250 may alternatively performany one or more of the acts shown in FIG. 2 as being performed byexternal computer 275, including performing automated speech recognitionon a user-specified query, identifying the type(s) of information theuser seeks and/or action(s) the user wishes to perform (e.g., byperforming example process 1200 of FIG. 12), generating search queriesto be submitted to various search engines, and/or other acts.Embodiments of the invention are not limited to being implemented usingany particular architecture.

Another example of a mode of processing in which an external computerfacilitates a search by a client device is depicted in FIG. 8. In thisexample, an application program executing on client device 803 receivesa search query from a user 801, which may be in text format, audioformat, or some other format. Client device 803 passes the user-suppliedsearch query to server 805, which determines which search engines toquery, generates URLs for the determined search engines with theappropriate search strings, and returns the URLs to the applicationprogram on client device 803. However, in other embodiments, thedetermination as to which search engines to query may be made on theclient device, and/or the client device may generate URLs to be usedwith different search engines. In embodiments in which the determinationas to which search engines to query is made on the server, the servermay instruct the client device which search engines to query and theclient device may generate the URLs for these search engines.

As noted above, in situations wherein the user-specified query is avoice query provided as audio data, server 805 may, in some embodiments,perform automated speech recognition on the audio data to generate arecognition result and determine which search engines to query based onthe recognition result. In other embodiments, automated speechrecognition may be performed on client device 803, and the recognitionresult, rather than the audio data, may be sent from client device 803to server 805. Other configurations are possible, including ones inwhich ASR operations are distributed in any suitable manner between theclient device and the server.

In some embodiments, the determination of which search engines to queryusing any of the above-discussed criteria may be made by the applicationprogram executing on the client device. In other embodiments, thedetermination of which search engines to query using the above-discussedcriteria may be made by a computer (e.g., server 805 in FIG. 8) externalto the client device executing the application program, and the externalcomputer may provide information identifying the search engines to bequeried to the application program. In other embodiments, thedetermination may be distributed between the client device and one ormore external computers in any suitable way. For example, theapplication program may determine which search engines to query usinginformation provided to it by the external computer to facilitate thedetermination.

III. Issuing Queries to Selected Search Engines

Referring again to FIG. 2, after client device 250 receives information(e.g., instructions) generated by external computer 275 in act 213,example process 200 proceeds to act 215, wherein the client deviceprocesses the information. Depending on the nature of the informationreceived, this may take any of numerous forms.

For example, when the information includes search queries for multiplesearch engines, the processing can include issuing the search queries tothe set of search engines identified in act 1215. An example environment300 in which a client device issues search queries to each of aplurality of search engines is depicted in FIG. 3. In the example shown,user 301 accesses the internet 305 via a client device 303. Searchengines 307 a, 307 b, 307 c, . . . , and 307 n execute on servers thatare accessible via the Internet 305. As noted above, the informationprovided to the client device may include a search query that may, forexample, take the form of a URL formatted appropriately for submissionto a particular search engine. In some embodiments, the URL may includean encoded representation of the original user-specified query, and/ormay include any other suitable form(s) of information.

It should be appreciated that although the client device issuing searchqueries may have certain advantages, embodiments of the invention arenot limited to such an implementation. In this respect, issuing searchqueries from the client device may provide the advantage that the searchengines that are queried do not receive a large volume of queries fromthe same server or IP address. As explained above, if the server wereissuing queries to the search engine on behalf of a large number ofclients, the search engines being queried may receive a very largenumber of queries from a single source, and may perceive this behavioras a single user issuing an enormous number of search queries to thesearch engine. Many search engine operators would consider this behaviorto be an abuse of the search engine service and/or a potential denial ofservice attack and would take action to block search queries to theirsearch engines from a source issuing such a large number of queries. Byissuing the search queries from the client device, the search enginesmay perceive the queries as being issued from the client to device thatcaused them to be issued, and may treat the issuance of such queries asnormal user behavior.

However, embodiments of the invention are not limited to beingimplemented in this manner. For example, in embodiments in which theURLs or queries are generated by a computer external to the clientdevice, that external computer may instead issue the queries (or aportion of the queries), receive the search results from the searchengines to which queries were issued, and return search results to theclient device. Any of numerous implementations are possible.

In some embodiments, a search query may not be submitted to a searchengine, but rather to an API provided by a search engine designed toprovide content formatted for processing by one or more native controlson a client device. In this respect, the inventors have recognized thatcontent formatted for processing by native controls may, in someinstances, load faster and provide a superior user experience thanmarkup language (e.g., HTML) content supplied by some search engines inresponse to search queries. For example, an API made available by asearch engine may respond to a search query with results formatted inJavaScript, which native controls on a client device may be configuredto process to render a user display. Of course, embodiments of theinvention are not limited to employing JavaScript, as any suitablerepresentation enabling processing by native controls on a client devicemay alternatively be used.

As noted above, executing instructions in act 215 is not limited toissuing queries to one or more identified search engines. For example,act 215 may include initiating one or more native actions, specified inthe information received from the external computer, on client device250. Any suitable action(s) may be initiated and/or performed. Forexample, a native action may include the launch of an applicationresident on the client device, as described above.

As noted above, the information processed in act 215 may specify that anative action be initiated and/or performed in any suitable fashion. Forexample, instructions may specify that an application be executed insuch a manner that if it is stopped by the user, the client devicereturns immediately to the search application on the client device thatreceived the user's query, so that search results may be presented tothe user (e.g., as a backup option to the execution of the applicationin case the user's intent was not correctly informed). Alternatively,the instructions may specify presenting the search results along with anoption to launch a native application, as embodiments of the inventionthat relate to performing a native action may be implemented in anysuitable way.

The information processed in act 215 may specify that any of a set ofalternative native actions be initiated. For example, the informationmay specify that one or more of a collection of potential native actionsbe initiated, depending on the capabilities of the device. For example,rather than specifically identifying a particular media playerapplication to be launched, the information may specify a list of mediaplayer applications. The client device may determine whether any of themedia player applications in the list is installed on the device, and ifa named media player application is determined to be installed, thatapplication may be launched. If multiple of the listed media players arepresent, the information from the external computer may specify aprioritization scheme based on any suitable criteria, and the clientdevice may launch the highest priority media player that is resident, orthe client device may be free to select any of the media playersspecified. Thus, embodiments of the invention are not limited toemploying an ordered list, as any suitable technique for identifying oneor more native actions to be initiated and/or performed may be employed.

If an ordered list is used, the prioritization may be determined in anysuitable manner. For example, some embodiments of the invention providefor the prioritized order in which applications are named beinginfluenced by consideration received from one or more providers ofapplications. For example, a provider may pay to have its applicationnamed on a list provided by the external computer before a competitor'sapplication, so that its application will be launched if both reside onthe client device. If two application providers both providecompensation, the prioritization between them may be determined in anysuitable manner, such as by always giving preference to an applicationwhose provider paid greater compensation, by alternating whichapplication is prioritized in any suitable manner (e.g., in proportionto compensation paid by each), or in any other suitable manner. Thesetechniques are merely examples, as prioritization may be determined inany suitable manner.

IV. Receiving Search Results from Search Engines

At the completion of act 215, example process 200 proceeds to act 217,wherein results are received from each search engine to which a searchquery was submitted. Search results may be received via any suitablecommunications infrastructure and/or protocol, as embodiments of theinvention are not limited in this respect.

As noted above, search results may be formatted in any of a variety ofways. For example, some search engines may provide results as ahypertext markup language (HTML) web page including a list of hyperlinksto the content and/or web pages identified as relevant to the searchquery. Other search engines may provide results in a format (e.g.,JavaScript notation) which allows native controls on the device to beused to process the results for display. Other search engines mayprovide results in other formats. It should be appreciated that allsearch results received need not be in the same format, as multipleresult formats may be received and displayed.

It should also be appreciated that the example mode of processing shownin FIG. 2 (wherein client device 250 issues search queries, and receivesresults from the search engines to which the queries are submitted) ismerely one example, and that numerous other implementations arepossible. For example, in embodiments of the invention in which anexternal computer (e.g., server 605 in FIG. 6) issues search queries tosearch engines, each search engine may provide its results to theexternal computer, and the external computer may provide these resultsto the client device. Embodiments of the invention are not limited tobeing implemented in any particular way.

V. Displaying Search Results

At the completion of act 217, example process 200 proceeds to act 219,wherein results are displayed. The results may, for example, includesearch results generated by search engines to which search queries weresubmitted, the results of initiating a native action, and/or any othersuitable information. In some embodiments, results may be displayed on adisplay device that is integrated with or coupled to the client device.The results may be displayed in any of a variety of formats and may bepresented in any of a variety of possible ways.

In some embodiments, search results returned from search engines thatprovide search results as an HTML web page may be displayed by renderingthe HTML code in the web page. This is different from the displayedresults from metasearch engines such as the above-discussed Dogpile™metasearch engine, which do not render the HTML code received from asearch engine, but rather extract links from this HTML code and generatenew HTML code to display the extracted links.

For search results that are not received in HTML format, HTML code maybe automatically generated to display the received search results, andthese search results may be displayed by rendering the automaticallygenerated HTML code.

As noted above, the search engines for which results are displayed mayinclude one or more general-purpose search engines and/or one or moresite-specific search engines. As such, the search results displayed inresponse to one user-specified query may have been returned by each ofmultiple different search engines (one or more of which may be ageneral-purpose search engine, and one or more of which may be asite-specific search engine).

In some embodiments, the search results returned by each search enginemay be visually separated from search results returned by other searchengines, so that it is clear to the user viewing the search resultswhich search engine provided each set of search results. In addition, insome of such embodiments, the order of the list of search resultsreturned from each search engine may be preserved in the displayedsearch results for each search engine, such that the user may view howrelevant each search engine determined each search result to be in theresults returned.

Search results from each search engine may be visually separated in anyof a variety of possible ways. For example, search results from multiplesearch engines may be displayed concurrently on the display butseparated by columns, lines, or in some other way. In some embodiments,only one of the search engines' search results may be displayed on thedisplay at any given time, and the user may control which of the searchengines' search results are displayed (e.g., by selecting an icon,button, or other user interface element).

In some embodiments, search results may be displayed using a carouselmetaphor, in which the carousel has a plurality of slots, each slotcorresponding to one of the search engines from which search resultswere received. User selection of one of the slots of the carousel may,for example, cause results returned by the corresponding search engineto be displayed. FIGS. 10A and 10B show screenshots of example displays1000A-1000B which employ a carousel metaphor. Example displays1000A-1000B may, for example, be presented by the Dragon Search™application offered by Nuance Communications, Inc., of Burlington, Mass.for use with the iPhone™ device offered by Apple Computer, Inc. ofCupertino, Calif.

Example display 1000A shown in FIG. 10A presents results generated inresponse to a first user-specified query, and example display 1000Bshown in FIG. 10B presents results generated in response to a seconduser-specified query. Each of example displays 1000A-1000B includes acarousel having multiple slots, with each slot being assigned adifferent search engine or native action. Thus, in FIG. 10A, exampledisplay 1000A comprises carousel 1001A which includes slots 1020A,1025A, 1030A, 1035A and 1040A, each of which corresponds to a differentsearch engine or native action. In FIG. 10B, example display 1000Bcomprises carousel 1001B which includes slots 1020B, 1025B, 1030B, 1035Band 1040B, each of which corresponds to a different search engine ornative action. In the examples shown, the search engine or native actionto which each slot corresponds is identified via an icon that identifiesthe search engine or native action, although any suitable manner ofidentification (e.g., symbol, text, image, etc.) may be employed.

The examples shown in FIGS. 10A-10B depict search engines and/or nativeactions assigned to slots in a carousel that are chosen based at leastin part on the content of a user-specified query. In FIG. 10A, exampledisplay 1000A (FIG. 10A) shows via query field 1009A that theuser-specified query is “Legal Seafood Boston Mass.” A set of searchengines and/or native actions are chosen based at least in part on thecontent of this query, and assigned to each of slots 1020A, 1025A,1030A, 1035A and 1040A. As noted above, in some embodiments, choosingthe set of search engines and/or native actions may involve determiningone or more information categories to which the query relates, and thenemploying a predefined mapping (an example of which is shown in Table 1,above) to identify a set of search engines and/or native actions thatmake available information relevant to the identified one or moreinformation categories. In the example shown in FIG. 10A, theuser-specified query “Legal Seafood Boston Mass.” is associated with the“restaurants” and “location” information categories, so that a mapapplication is assigned to slot 1030A, a search engine offered by the“Yelp” restaurant review website is assigned to slot 1035A, a serviceproviding access to telephone numbers included in results generated by ageneral-purpose search engine is assigned to slot 1025A, a search engineoffered by the “Wikipedia” reference website is assigned to slot 1040A,and a search engine provided by the “Twitter” social networking websiteis assigned to slot 1020A.

In example display 1000B, shown in FIG. 10B, query field 1009A indicatesthat the user-specified query is “Avatar.” As in example display 1000A,a set of search engines and/or native actions is chosen based at leastin part on the content of this query, and assigned to slots 1020B,1025B, 1030B, 1035B and 1040B. In this example, the user-specified query“Avatar” is associated with the “Movies” and “Title [not in theaters]”information categories, so that the set assigned to slots 1020B, 1025B,1030B, 1035B and 1040A (as defined by the example mapping provided byTable 1) includes a search engine offered by the “IMDB” movieinformation website assigned to slot 1030B, a search engine offered bythe “rottentomatoes” movie review website assigned to slot 1035B, amedia player application assigned to slot 1025B, a search engine offeredby the “Wikipedia” reference website assigned to slot 1040B, and asearch engine provided by the “Twitter” social networking websiteassigned to slot 1020B.

It should be appreciated from the examples shown in FIGS. 10A-10B thatsome embodiments of the invention enable the slots in a carousel to beassigned to sets of search engines and/or native actions based on thecontent of a user-specified query, rather than the slots being assignedto a static set regardless of the query that is received. As such, theassignment of a search engine or native action to a carousel slot insome embodiments may be considered “dynamic” in that it is based atleast in part on the content of a user's query. It should beappreciated, however, that some embodiments of the invention may notassign all slots dynamically. For example, in some embodiments of theinvention, one or more search engines and/or native actions assigned toslots may be statically defined, and others may be dynamicallydetermined For example, one or more slots in a carousel may each beassigned a predefined search engine or native action, and one or moreother slots may each be assigned dynamically. The assignment of a searchengine or native action to a slot in a carousel may be performed in anysuitable way, as embodiments of the invention are not limited to anyparticular manner of implementation.

Example display 1000A includes area 1005A in which results generated bya selected search engine or native action are displayed. In the exampleshown, a user selects a search engine or native action to cause itsresults to be displayed in area 1005A by providing input to (e.g.,tapping if a client device includes a touch display, and/or providingany other suitable form of input) a corresponding icon shown in acarousel slot. In this respect, example display 1000A includes indicator1007A to indicate the slot in the carousel 1001A selected for display.In the example shown, the indicator 1007A indicates that resultsgenerated by the map application identified by the icon in slot 1030Aare displayed in area 1005A. In FIG. 10B, indicator 1007B indicates thatresults generated by the search engine provided by the “IMDB” websiteidentified by the icon shown in slot 1030B are displayed in area 1005B.

In some embodiments, the placement of particular search engines ornative actions in particular carousel slots may be determineddynamically (e.g., based on an estimated similarity between the contentof the query (and/or other information) and content made available by asearch engine or native action). For example, some embodiments of theinvention may provide for results generated by search engines providingaccess to a specific type of information that it is believed the userseeks to be presented more prominently than results generated by searchengines providing access to more general types of information. Forexample, a most aggressive specific search engine option (e.g., providedby a site that makes available specialized information relating to theuser-specified query) may be presented most prominently, and other lessaggressive specific search engine options (e.g., providing lessspecialized information, but potentially encompassing the type ofinformation the user seeks) may be presented less prominently.

Displaying some information more prominently than other information maybe accomplished in any of numerous ways. In the examples shown in FIGS.10A-10B, information generated by a particular search engine or nativeapplication may be displayed more prominently than other information byallocating that information to a slot in the carousel that has a moreprominent appearance than others. For example, the slot may be one whichis displayed upon the search results initially being presented so thatthe user need take no action (e.g., scrolling the carousel) to viewthem, one which is justified with respect to the display (e.g.,centered, left-justified, right-justified), and/or defined using anyother criteria. In the example display 1000B shown in FIG. 10B, resultsgenerated by the “IMDB” search engine, identified as those to bedisplayed most prominently, are shown in slot 1030B, which is centeredwith respect to the display. In addition, in this example, slot 1030B isshown immediately upon results being presented, as indicated byindicator 1007B. In the example display 1000A shown in FIG. 10A,information generated by the map application, identified as that whichis to be displayed most prominently, are shown in slot 1030A, which iscentered with respect to the display and shown immediately as indicatedby indicator 1007A.

In some embodiments (e.g., those in which a user may specify a searchengine to be queried), a user may configure which search engines are tobe allocated to which slots in the carousel. In addition, in someembodiments, the allocation of carousel slots to particular searchengines may be made based on the results returned by the search engines.For example, if one of the search engines selected at act 203 of process200 as a search engine to be queried does not return any search resultsor returns an indication that no search results were found as a resultof the query, a slot in the carousel may not be allocated for thatsearch engine, and the page returned from that search engine may not bedisplayed to the user.

Not all of the results returned from a particular search engine inresponse to a query may fit in area 1005 at once. Thus, in someembodiments, a user may be provided with the capability to scroll up anddown the list of search results and/or adjust the zoom level at whichresults are displayed so as to adjust which portion of the searchresults are displayed in area 1005.

As shown in FIGS. 10A-10B, in some embodiments, not all of the carouselslots may be able to be displayed at one time, owing to the size of thedisplay. For example, in FIG. 10A, slots 1020A and 1040A are onlypartially displayed. In addition, carousel 1001A may include slots thatare not displayed at all in example display 1000A. As such, someembodiments of the invention enable a user to adjust which slots of thecarousel are displayed by shifting the carousel to the right side and/orto the left side of the display. This may be accomplished in any of avariety of ways. For example, the user may drag the carousel in onedirection using a pointer device such as a mouse, or drag the carouselin one direction using his or her finger in embodiments in which display1000 is presented on a touch screen.

In some embodiments, a carousel 1001 may wrap around a display 1000. Forexample, if the user continuously drags carousel 1001A to the left sideof display 1000A, then the slots of carousel 1001A moving the left willmove off of the left side of the display and will eventually re-enterthe display on the right side. In this respect, dragging carousel 1001Ain a particular direction can be thought of as rotating the carousel inthat direction.

As noted above, example displays 1000A and 1000B each include a queryfield 1009 (i.e., example display 1000A includes query field 1009A, andexample display 1000B includes query field 1009B). In some embodiments,query field 1009 indicates to a user the query that is the basis forinformation supplied to a selected search engine or native applicationfor which information is displayed in area 1005. In this respect, itshould be appreciated from the description above that, in someembodiments, different representations of a user-specified query may besubmitted to different facilities. As such, query field 1009 may displaythe user-supplied query that is the basis for information submitted toeach different search engine or native application in response to thatquery. Alternatively, the query shown in the field may change with theselection of different slots on the carousel, to show the informationsubmitted to each.

A user may input a search query into field 1009 in any of a varietyways. In some embodiments, a user may provide text input to field 1009(e.g., via a physical or touch screen keyboard). In some embodiments, auser may provide voice input to field 1009. In the examples of FIGS.10A-10B, when a user wishes to supply voice input to field 1009, theuser may select (e.g., touch) button 1011 and speak his or her searchquery into a microphone on the client device.

In some embodiments, queries for each of the search engines selected inact 203 may be issued automatically in response to receiving the searchquery from the user in act 201. In other embodiments, the queries forall or some of the search engines selected in act 203 may not be issuedto those search engines until the user provides input requesting to viewthe search results from those search engines. For example, the searchresults from some search engines may include a large amount of content.As one example, the search results from a search engine for a web sitethat hosts video clips may include a thumbnail image of each videoincluded in the list of results. Consequently, transferring the searchresults from the search engine to the client device or external computermay take a relatively long time. The time taken to transfer the searchresults may be time that is not well spent if the user does not everview the search results from that search engine. Thus, in someembodiments, search queries to search engines that return resultsincluding bandwidth-intensive content, such as a large number of images,video data, audio data, and/or other bandwidth-intensive content may notbe issued automatically in response to receiving a user-supplied searchquery, while search queries to search engines that return results thatdo not include bandwidth-intensive content may be issued automatically.

It should be appreciated that the size of the content (or the expectedsize of the content) of a set of search results is one example of acriterion that may be used to determine whether to issue a search queryautomatically to a particular search engine or whether to await someuser indication before issuing the query to that search engine. Any of avariety of other criteria or combinations of criteria may be used.

Some embodiments of the invention may provide for displaying sponsoredcontent and/or other information in response to a user-specified query.For example, in accordance with some embodiments of the invention, inresponse to a query being received, a search engine is identified towhich to submit a representation of the query. The identifying may bebased at least in part on consideration received from an operator of theat least one search engine.

The display of sponsored content may be implemented in any of numerousways. For example, some embodiments may enable a banner which includessponsored content on a display. In other embodiments, sponsored contentmay be displayed within a particular slot on a carousel. In yet otherembodiments, the order in which icons are arranged in slots on thecarousel may be influenced by consideration received an entity. Forexample, a search engine provider may pay to have its results displayedmore to prominently than other search engines' results, such as bydisplaying the search engine's results in the center slot of thecarousel, or by having its results be shown immediately upon any resultsbeing made available. The display of sponsored content may beimplemented in any of numerous ways, as embodiments of the invention arenot limited in this respect.

The inventors have recognized that the type of information a user seeksmay be indicative of promotional offers to which the user may bereceptive. Thus, in some embodiments, sponsored content may be driven bythe content of a user's query. For example, if a user's query indicatesthat he/she seeks information on a particular restaurant of a particularcategory of restaurant, then another restaurant in that category may payto have one of the slots in the carousel show information on itslocation(s), reviews, coupons or other promotional offer.

Alternatively, sponsored content may be driven by information other thanthe content of a user's query. For example, sponsored content may bedriven by such auxiliary information as the location of a client device,previous searches and/or browsing activity performed by the user,demographic information on the user, the speed at which the device ismoving (which may indicate whether and what type of vehicle the deviceis travelling in), and/or other information. For example, if auser-specified query indicates that the user is looking for informationon nearby restaurants, then a restaurant located close to the user'scurrent location may have information presented on the display based, atleast in part, on the restaurant having paid to have its informationdisplayed to such users. Any of numerous implementations are possible.

VI. Monitoring User Actions

As discussed above, when search results from one or more search enginesare displayed to a user, the user may view some or all of the lists ofsearch results, and may access hyperlinks to view web pages or contentidentified in the lists of results.

In some embodiments, the application program may monitor user actionstaken with respect to the search results and may store information aboutthe monitored user actions. The monitored actions can take any suitableform, as the aspect of the invention relating to monitoring user actionsis not limited in this respect. In some embodiments, the monitored useractions may include, for example, which search engines' search resultsthe user viewed and what web pages and/or content the user accessed.

The information describing the monitored user actions may besubsequently used to update language models used in automatic speechrecognition of voice queries, determine which search engines to query inresponse to subsequent user queries, determine how search engine resultsare to be presented (e.g., determine which carousel slot is the defaultslot and/or in what order the search engines are to be positioned in thecarousel slots), and/or to facilitate various other aspects of queryingsearch engines, displaying search engine results and/or launching nativebehaviors.

In embodiments in which a computer external to the client device usessuch information to facilitate any of the above-discussed aspects ofquerying a search engine or launching native behavior, the client devicemay send the monitored information to the external computer. Theexternal computer may associate this information with the client device,the user of the client device, and/or the network address from which theinformation was sent, and may store it for subsequent use.

VII. Additional Implementation Detail

The above discussed computing devices (e.g., client devices, servers,external computers, and/or any other above-discussed computing devices)may be implemented in any of a variety of ways. FIG. 11 is a blockdiagram an illustrative computing device 1100 that may be used toimplement any of the above-discussed computing devices.

The computing device 1100 may include one or more processors 1101 andone or more tangible, non-transitory computer-readable storage media(e.g., memory 1103). Memory 1103 may store, in tangible non-transitorycomputer-readable storage media computer instructions that implement anyof the above-described functionality. Processor(s) 1101 may be coupledto memory 1103 and may execute such computer instructions to cause thefunctionality to be realized and performed. Computing device 1100 mayalso include a network input/output (I/O) interface 1105 via which thecomputing device may communicate with other computers (e.g., over anetwork), and may also include one or more user I/O interfaces, viawhich the computer may provide output to and receive input from a user.The user I/O interfaces may include devices such as a keyboard, a mouse,a microphone, a display device (e.g., a monitor or touch screen),speakers, a camera, and/or various other types of I/O devices.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. It should beappreciated that any component or collection of components that performthe functions described above can be generically considered as one ormore controllers that control the above-discussed functions. The one ormore controllers can be implemented in numerous ways, such as withdedicated hardware, or with general purpose hardware (e.g., one or moreprocessors) that is programmed using microcode or software to performthe functions recited above.

In this respect, it should be appreciated that one implementation ofvarious embodiments of the present invention comprises at least onetangible, non-transitory computer-readable storage medium (e.g., acomputer memory, a floppy disk, a compact disk, and optical disk, amagnetic tape, a flash memory, circuit configurations in FieldProgrammable Gate Arrays or other semiconductor devices, etc.) encodedwith one or more computer programs (i.e., a plurality of instructions)that, when executed on one or more computers or other processors,performs the above-discussed functions of various embodiments of thepresent invention. The computer-readable storage medium can betransportable such that the program(s) stored thereon can be loaded ontoany computer resource to implement various aspects of the presentinvention discussed herein. In addition, it should be appreciated thatthe reference to a computer program which, when executed, performs theabove-discussed functions, is not limited to an application programrunning on a host computer. Rather, the term computer program is usedherein in a generic sense to reference any type of computer code (e.g.,software or microcode) that can be employed to program a processor toimplement the above-discussed aspects of the present invention.

Various aspects of the present invention may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the to foregoing and are therefore notlimited in their application to the details and arrangement ofcomponents set forth in the foregoing description or illustrated in thedrawings. For example, aspects described in one embodiment may becombined in any manner with aspects described in other embodiments.

Also, embodiments of the invention may be implemented as one or moremethods, of which an example has been provided. The acts performed aspart of the method(s) may be ordered in any suitable way. Accordingly,embodiments may be constructed in which acts are performed in an orderdifferent than illustrated, which may include performing some actssimultaneously, even though shown as sequential acts in illustrativeembodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed. Such terms areused merely as labels to distinguish one claim element having a certainname from another element having a same name (but for use of the ordinalterm).

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” “having,” “containing”, “involving”, andvariations thereof, is meant to encompass the items listed thereafterand additional items.

Having described several embodiments of the invention in detail, variousmodifications and improvements will readily occur to those skilled inthe art. Such modifications and improvements are intended to be withinthe spirit and scope of the invention. Accordingly, the foregoingdescription is by way of example only, and is not intended as limiting.The invention is limited only as defined by the following claims and theequivalents thereto.

What is claimed is:
 1. A method, performed by at least one computer,comprising acts of: (A) receiving a query from a device and locationdata indicating a location of the device, the location data having alevel of specificity; (B) in response to the query being received,identifying at least one first search engine to which to submit arepresentation of the query and information indicating the location ofthe device; (C) determining whether the level of specificity of thelocation data received in (A) is sufficient for the at least one firstsearch engine; (D) when the level of specificity of the location data issufficient, instructing the device to issue the representation of thequery to the at least one first search engine; and (E) when the level ofspecificity of the location data is not sufficient, instructing thedevice to send, to the at least one computer, location data at a greaterlevel of specificity, wherein the act (C) comprises determining that thelevel of specificity of the location data received in (A) is notsufficient for the at least one first search engine, and wherein themethod further comprises acts of: (F) receiving location data at thegreater level of specificity; and (G) instructing the device to submit arepresentation of the query and information specifying the location ofthe device at the greater level of specificity to the at least one firstsearch engine.
 2. The method of claim 1, wherein the at least one firstsearch engine comprises a site-specific search engine.
 3. The method ofclaim 1, wherein the act (C) is performed based at least in part oncontent of the query.
 4. The method of claim 1, wherein the act (C) isperformed based at least in part on predefined specificity requirementsof the at least one first search engine.
 5. At least one non-transitorycomputer readable medium having encoded thereon instructions which, whenexecuted by at least one computer, perform a method comprising acts of:(A) receiving a query from a device and location data indicating alocation of the device, the location data having a level of specificity;(B) in response to the query being received, identifying at least onefirst search engine to which to submit a representation of the query andinformation indicating the location of the device; (C) determiningwhether the level of specificity of the location data received in (A) issufficient for the at least one first search engine; (D) when the levelof specificity of the location data is sufficient, instructing thedevice to issue the representation of the query to the at least onefirst search engine; and (E) when the level of specificity of thelocation data is not sufficient, instructing the device to send, to theat least one computer, location data at a greater level of specificity,wherein the act (C) comprises determining that the level of specificityof the location data received in (A) is not sufficient for the at leastone first search engine, and wherein the method further comprises actsof: (F) receiving location data at the greater level of specificity; and(G) instructing the device to submit a representation of the query andinformation specifying the location of the device at the greater levelof specificity to the at least one first search engine.
 6. The at leastone non-transitory computer readable medium of claim 5, wherein the atleast one first search engine comprises a site-specific search engine.7. The at least one non-transitory computer readable medium of claim 5,wherein the act (C) is performed based at least in part on content ofthe query.
 8. The at least one non-transitory computer readable mediumof claim 5, wherein the act (C) is performed based at least in part onpredefined specificity requirements of the at least one first searchengine.
 9. At least one computer, comprising: at least one hardwareprocessor; and at least one memory storing processor-executableinstructions that, when executed by the at least one hardware processor,cause the at least one hardware processor to: receive a query from adevice and location data indicating a location of the device, thelocation data having a level of specificity; in response to the querybeing received, identify at least one first search engine to which tosubmit a representation of the query and information indicating thelocation of the device; determine whether the level of specificity ofthe received location data is sufficient for the at least one firstsearch engine; when the level of specificity of the location data issufficient, instruct the device to issue the representation of the queryto the at least one first search engine; and when the level ofspecificity of the location data is not sufficient, instruct the deviceto send, to the at least one computer, location data at a greater levelof specificity, wherein the determining comprises determining that thelevel of specificity of the received location data is not sufficient forthe at least one first search engine, and wherein the at least onehardware processor is programmed to receive the location data at thegreater level of specificity and to instruct the device to submit arepresentation of the query and information specifying the location ofthe device at the greater level of specificity to the at least one firstsearch engine.
 10. The at least one computer of claim 9, wherein the atleast one first search engine comprises a site-specific search engine.11. The at least one computer of claim 9, wherein the determining isperformed based at least in part on content of the query.
 12. The atleast one computer of claim 9, wherein the determining is performedbased at least in part on predefined specificity requirements of the atleast one first search engine.